2024-03-25T16:28:27.380 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:28:27.380 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283 --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --verbose --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/orig.config.yaml 2024-03-25T16:28:27.424 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:28:27.508 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621283' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK07BMq4oLcI6wwv4DMUGtscYrh9LL59zIyHfosL9Nj8Ex27kI0DSaI4yISwPX7nrZdnKhC/CUVAG8gzxGw/l/A= smithi133.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqMAS6+p/Y9IuVWSK3DNUKE1ozDOdJ65gM1d/iplZtXgOL9VzTNrOGN+qGVjwh4Pt4gCuRRJXRbtcLOhe0ylkA= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:28:27.508 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:28:27.509 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:28:27.509 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:28:27.510 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:28:27.534 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:28:27.534 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:28:27.534 INFO:teuthology.packaging:ref: None 2024-03-25T16:28:27.534 INFO:teuthology.packaging:tag: None 2024-03-25T16:28:27.534 INFO:teuthology.packaging:branch: main 2024-03-25T16:28:27.534 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:28:27.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:28:28.060 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:28:28.062 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:28:28.067 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:28:28.067 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:28:28.079 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:28:28.095 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:28:28.101 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:28:28.126 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:23:19.371011', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK07BMq4oLcI6wwv4DMUGtscYrh9LL59zIyHfosL9Nj8Ex27kI0DSaI4yISwPX7nrZdnKhC/CUVAG8gzxGw/l/A='} 2024-03-25T16:28:28.145 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:23:19.372598', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqMAS6+p/Y9IuVWSK3DNUKE1ozDOdJ65gM1d/iplZtXgOL9VzTNrOGN+qGVjwh4Pt4gCuRRJXRbtcLOhe0ylkA='} 2024-03-25T16:28:28.145 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:28:28.151 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:28:28.151 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:28:28.152 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:28:28.227 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8a07fc40d0>, signals=[15]) 2024-03-25T16:28:28.227 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:28:28.233 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:28:28.233 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T16:28:28.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:28.341 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T16:28:28.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:28.436 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:28:28.444 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2024-03-25T16:28:28.461 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2024-03-25T16:28:28.461 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2024-03-25T16:28:28.518 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="9" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="9" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:28:28.519 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:28:28.520 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:28:28.520 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2024-03-25T16:28:28.539 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2024-03-25T16:28:28.555 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2024-03-25T16:28:28.556 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2024-03-25T16:28:28.611 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="CentOS Stream" 2024-03-25T16:28:28.611 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="9" 2024-03-25T16:28:28.611 INFO:teuthology.orchestra.run.smithi133.stdout:ID="centos" 2024-03-25T16:28:28.611 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:28:28.611 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="9" 2024-03-25T16:28:28.611 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:28:28.611 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:28:28.612 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2024-03-25T16:28:28.612 INFO:teuthology.orchestra.run.smithi133.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:28:28.612 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:28:28.612 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:28:28.612 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:28:28.612 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:28:28.612 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:28:28.612 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2024-03-25T16:28:28.639 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:28:28.649 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:28:28.656 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:28:28.656 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:28:28.658 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:28:28.673 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:28:28.679 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:28:28.680 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:28:28.716 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:28:28.739 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:28:28.795 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:28:28.813 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:28:28.813 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:28:28.813 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:28:28.814 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:28:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:28:28.829 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2024-03-25T16:28:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:28:28.832 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-25T16:28:28.885 INFO:teuthology.orchestra.run.smithi133.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:28:28.885 INFO:teuthology.task.kernel:Running kernel on smithi133: 5.14.0-361.el9.x86_64 2024-03-25T16:28:28.885 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum install -y kernel 2024-03-25T16:28:28.890 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:28:28.890 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.14.0-361.el9.x86_64 2024-03-25T16:28:28.890 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2024-03-25T16:28:31.220 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - AppStream 9.2 MB/s | 6.7 MB 00:00 2024-03-25T16:28:31.317 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 9.0 MB/s | 6.7 MB 00:00 2024-03-25T16:28:32.892 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-25T16:28:32.954 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-25T16:28:34.053 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-25T16:28:36.049 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-03-25T16:28:36.106 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-25T16:28:40.129 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-25T16:28:41.605 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-25T16:28:45.198 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 5.9 MB/s | 21 MB 00:03 2024-03-25T16:28:50.114 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-25T16:28:52.162 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:28:52.163 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:28:52.304 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:28:52.315 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:28:52.316 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:28:52.316 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:28:52.316 INFO:teuthology.orchestra.run.smithi133.stdout:Install 4 Packages 2024-03-25T16:28:52.316 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:28:52.316 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 76 M 2024-03-25T16:28:52.316 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 123 M 2024-03-25T16:28:52.316 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:28:52.440 INFO:teuthology.orchestra.run.smithi133.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 533 kB/s | 44 kB 00:00 2024-03-25T16:28:53.041 INFO:teuthology.orchestra.run.smithi133.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-25T16:28:53.657 INFO:teuthology.orchestra.run.smithi133.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-25T16:28:53.908 INFO:teuthology.orchestra.run.smithi133.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 19 MB/s | 27 MB 00:01 2024-03-25T16:28:53.908 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:53.908 INFO:teuthology.orchestra.run.smithi133.stdout:Total 48 MB/s | 76 MB 00:01 2024-03-25T16:28:54.574 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:28:54.955 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:28:54.955 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:28:55.948 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:28:55.948 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:28:58.425 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:28:59.053 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:28:59.140 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:28:59.590 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 859 kB/s | 19 MB 00:22 2024-03-25T16:29:01.862 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:29:01.992 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:29:03.865 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-25T16:29:05.316 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-25T16:29:06.618 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-25T16:29:06.709 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:29:06.741 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:29:11.444 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:29:11.759 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-25T16:29:13.941 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:29:13.941 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:29:14.038 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:29:14.046 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:29:14.046 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout:Install 4 Packages 2024-03-25T16:29:14.047 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:29:14.048 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 76 M 2024-03-25T16:29:14.048 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 123 M 2024-03-25T16:29:14.048 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:29:14.511 INFO:teuthology.orchestra.run.smithi055.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 131 kB/s | 44 kB 00:00 2024-03-25T16:29:22.027 INFO:teuthology.orchestra.run.smithi055.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.9 MB/s | 15 MB 00:07 2024-03-25T16:29:27.793 INFO:teuthology.orchestra.run.smithi055.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 2.1 MB/s | 27 MB 00:13 2024-03-25T16:29:30.293 INFO:teuthology.orchestra.run.smithi055.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 2.1 MB/s | 33 MB 00:16 2024-03-25T16:29:30.295 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:30.295 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.7 MB/s | 76 MB 00:16 2024-03-25T16:29:30.983 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:29:31.429 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:29:31.429 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:29:32.447 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:29:32.447 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:29:35.071 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:29:35.698 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:29:35.811 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:29:38.677 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:29:38.815 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:29:40.880 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:29:40.880 INFO:teuthology.orchestra.run.smithi133.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:29:40.880 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:29:40.924 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:29:41.257 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:29:41.258 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:29:41.258 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:29:41.258 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:29:41.692 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:29:41.693 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:29:41.693 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:29:41.693 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:29:41.693 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:29:41.693 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:29:41.693 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:29:41.693 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:29:41.693 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:29:42.015 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:29:42.129 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:29:42.130 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:29:42.130 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:29:42.130 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:29:42.130 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:29:42.130 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:29:42.130 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:29:42.130 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:29:42.210 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:29:42.210 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:29:42.211 DEBUG:teuthology.orchestra.run.smithi133:> sudo rpm -qi grub2-tools 2024-03-25T16:29:42.248 INFO:teuthology.orchestra.run.smithi133.stdout:Name : grub2-tools 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Epoch : 1 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Version : 2.06 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Release : 68.el9 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: x86_64 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Group : Unspecified 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Size : 8264254 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:License : GPLv3+ 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Packager : builder@centos.org 2024-03-25T16:29:42.249 INFO:teuthology.orchestra.run.smithi133.stdout:Vendor : CentOS 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout:Summary : Support tools for GRUB. 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout:Description : 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout:hardware devices. 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:29:42.250 INFO:teuthology.orchestra.run.smithi133.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:29:42.251 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:29:42.251 INFO:teuthology.task.kernel:Updating grub on smithi133 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:29:42.251 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:29:42.862 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2024-03-25T16:29:43.892 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:29:43.932 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:29:44.260 INFO:teuthology.orchestra.run.smithi133.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:29:44.281 INFO:teuthology.orchestra.run.smithi133.stderr:done 2024-03-25T16:29:44.283 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2024-03-25T16:29:44.298 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.lKxpYVMDz6 2024-03-25T16:29:44.298 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lKxpYVMDz6 2024-03-25T16:29:44.366 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.lKxpYVMDz6 2024-03-25T16:29:44.537 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.lKxpYVMDz6 is 6KB 2024-03-25T16:29:44.585 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.lKxpYVMDz6 2024-03-25T16:29:44.599 DEBUG:teuthology.orchestra.run.smithi133:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:29:44.664 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:29:44.664 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:29:44.665 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:29:44.665 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:29:44.666 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:29:44.823 DEBUG:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2024-03-25T16:29:49.028 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:30:14.826 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:30:14.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-25T16:30:14.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:30:19.777 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:30:19.778 INFO:teuthology.orchestra.run.smithi055.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:30:19.778 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:30:19.822 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:30:20.177 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:30:20.177 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:30:20.177 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:30:20.177 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:30:20.619 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:30:20.620 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:30:20.620 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:30:20.620 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:30:20.620 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:30:20.620 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:30:20.620 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:30:20.620 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:30:20.620 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:30:20.919 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:30:21.022 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:30:21.023 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:30:21.023 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:30:21.023 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:30:21.023 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:30:21.024 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:30:21.024 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:30:21.024 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:30:21.106 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:30:21.107 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:30:21.107 DEBUG:teuthology.orchestra.run.smithi055:> sudo rpm -qi grub2-tools 2024-03-25T16:30:21.144 INFO:teuthology.orchestra.run.smithi055.stdout:Name : grub2-tools 2024-03-25T16:30:21.144 INFO:teuthology.orchestra.run.smithi055.stdout:Epoch : 1 2024-03-25T16:30:21.144 INFO:teuthology.orchestra.run.smithi055.stdout:Version : 2.06 2024-03-25T16:30:21.144 INFO:teuthology.orchestra.run.smithi055.stdout:Release : 68.el9 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: x86_64 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Group : Unspecified 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Size : 8264254 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:License : GPLv3+ 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Packager : builder@centos.org 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Vendor : CentOS 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Summary : Support tools for GRUB. 2024-03-25T16:30:21.145 INFO:teuthology.orchestra.run.smithi055.stdout:Description : 2024-03-25T16:30:21.146 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:30:21.146 INFO:teuthology.orchestra.run.smithi055.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:30:21.146 INFO:teuthology.orchestra.run.smithi055.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:30:21.146 INFO:teuthology.orchestra.run.smithi055.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:30:21.146 INFO:teuthology.orchestra.run.smithi055.stdout:hardware devices. 2024-03-25T16:30:21.146 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:30:21.146 INFO:teuthology.orchestra.run.smithi055.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:30:21.147 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:30:21.147 INFO:teuthology.task.kernel:Updating grub on smithi055 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:30:21.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:30:21.771 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2024-03-25T16:30:23.104 INFO:teuthology.orchestra.run.smithi055.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:30:23.141 INFO:teuthology.orchestra.run.smithi055.stderr:done 2024-03-25T16:30:23.143 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-25T16:30:23.159 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.JPCPQqQcoB 2024-03-25T16:30:23.159 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JPCPQqQcoB 2024-03-25T16:30:23.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.JPCPQqQcoB 2024-03-25T16:30:23.381 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.JPCPQqQcoB is 6KB 2024-03-25T16:30:23.430 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.JPCPQqQcoB 2024-03-25T16:30:23.445 DEBUG:teuthology.orchestra.run.smithi055:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:30:23.513 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:30:23.513 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:30:23.514 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:30:23.514 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:30:23.515 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:30:23.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2024-03-25T16:30:49.375 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-25T16:30:53.689 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:30:53.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-25T16:30:53.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:30:58.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-25T16:30:58.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:31:12.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2024-03-25T16:31:16.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-25T16:31:21.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-25T16:31:21.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:31:28.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-25T16:31:28.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:31:29.130 DEBUG:teuthology.orchestra.run.smithi133:> true 2024-03-25T16:31:29.542 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-25T16:31:29.543 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:31:29.543 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2024-03-25T16:31:29.557 INFO:teuthology.orchestra.run.smithi133.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:31:29.557 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:31:29.557 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:31:29.557 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:31:30.558 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:31:30.558 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:31:30.629 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2024-03-25T16:31:30.670 DEBUG:teuthology.parallel:result is None 2024-03-25T16:31:39.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2024-03-25T16:31:51.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-25T16:31:51.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:31:51.817 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-25T16:31:52.257 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-25T16:31:52.257 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:31:52.258 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2024-03-25T16:31:52.273 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:31:52.274 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:31:52.274 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:31:52.274 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:31:53.275 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:31:53.275 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:31:53.360 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2024-03-25T16:31:53.411 DEBUG:teuthology.parallel:result is None 2024-03-25T16:31:53.411 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:31:53.428 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:31:53.429 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:31:53.432 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:31:53.452 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:31:53.460 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:31:53.472 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:31:53.472 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:31:53.488 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:31:53.529 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:31:53.561 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:31:53.561 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:31:53.563 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:31:53.597 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:31:53.597 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:31:53.610 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:31:53.611 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:31:53.612 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:31:53.656 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:31:53.656 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:31:53.659 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:31:53.688 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:31:53.777 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:31:53.777 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:31:53.784 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:31:53.801 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:31:53.882 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:31:53.923 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:31:54.006 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:31:54.048 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T16:31:54.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:31:54.115 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:31:54.154 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:31:54.219 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:31:54.262 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:31:54.307 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T16:31:54.307 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:31:54.375 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2024-03-25T16:31:54.377 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2024-03-25T16:31:54.425 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:31:54.467 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:31:54.731 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:31:54.740 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:31:54.740 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:31:54.750 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:31:54.758 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2024-03-25T16:31:54.811 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2024-03-25T16:31:54.813 DEBUG:teuthology.orchestra.run.smithi133:> sudo service auditd rotate 2024-03-25T16:31:54.859 INFO:teuthology.orchestra.run.smithi133.stdout:Rotating logs: 2024-03-25T16:31:54.861 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:31:54.861 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2024-03-25T16:31:54.880 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2024-03-25T16:31:54.880 DEBUG:teuthology.orchestra.run.smithi133:> /usr/sbin/getenforce 2024-03-25T16:31:54.919 INFO:teuthology.orchestra.run.smithi133.stdout:Permissive 2024-03-25T16:31:54.919 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi133.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:31:54.920 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:31:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:31:54.949 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:31:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:31:54.987 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:31:54.987 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:31:55.023 DEBUG:teuthology.orchestra.run.smithi133:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:31:55.061 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:31:55.071 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T16:31:55.076 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T16:31:55.230 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:31:55.246 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:31:55.247 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi133.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:36:12.217 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi133.front.sepia.ceph.com')] 2024-03-25T16:36:12.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-25T16:36:12.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:36:12.310 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-03-25T16:36:12.393 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-03-25T16:36:12.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-25T16:36:12.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:36:12.481 DEBUG:teuthology.orchestra.run.smithi133:> true 2024-03-25T16:36:12.556 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-25T16:36:12.557 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:36:12.571 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:36:12.571 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:36:12.571 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:36:12.574 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:36:12.574 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:36:12.604 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:36:12.621 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:36:12.626 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:36:12.643 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:36:12.651 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2024-03-25T16:36:12.665 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2024-03-25T16:36:12.672 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: ntpd: command not found 2024-03-25T16:36:12.682 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:36:12.685 INFO:teuthology.orchestra.run.smithi133.stdout:506 Cannot talk to daemon 2024-03-25T16:36:12.698 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:36:12.702 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:36:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:36:12.757 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:36:12.761 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:36:12.761 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-03-25T16:36:12.761 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:36:12.761 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:36:12.761 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:36:12.761 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:36:12.768 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:36:12.771 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:36:12.771 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2024-03-25T16:36:12.771 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:36:12.771 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:36:12.772 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:36:12.772 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:36:12.772 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:36:12.780 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:36:12.780 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:36:12.780 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:36:12.780 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:36:12.786 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:36:12.786 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:36:12.786 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:36:12.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:36:12.789 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:36:12.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:36:13.012 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:36:13.012 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:36:13.016 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:36:13.016 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:36:13.357 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:36:13.357 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T16:36:13.357 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:36:13.386 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:36:13.387 DEBUG:teuthology.orchestra.run.smithi133:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:36:13.421 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:36:13.421 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T16:36:13.421 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:36:13.452 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:36:13.452 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:36:13.455 DEBUG:teuthology.orchestra.run.smithi133:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:36:13.522 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:36:13.535 DEBUG:teuthology.orchestra.run.smithi133:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:36:13.602 INFO:teuthology.orchestra.run.smithi133.stdout:check_obsoletes = 1 2024-03-25T16:36:13.603 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum clean all 2024-03-25T16:36:13.605 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:36:13.673 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2024-03-25T16:36:13.675 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2024-03-25T16:36:13.850 INFO:teuthology.orchestra.run.smithi133.stdout:74 files removed 2024-03-25T16:36:13.877 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw 2024-03-25T16:36:13.933 INFO:teuthology.orchestra.run.smithi055.stdout:74 files removed 2024-03-25T16:36:13.960 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw 2024-03-25T16:36:15.187 INFO:teuthology.orchestra.run.smithi133.stdout:ceph packages for x86_64 81 kB/s | 82 kB 00:01 2024-03-25T16:36:15.568 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 63 kB/s | 82 kB 00:01 2024-03-25T16:36:15.946 INFO:teuthology.orchestra.run.smithi133.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T16:36:16.477 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:36:16.738 INFO:teuthology.orchestra.run.smithi133.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-25T16:36:17.141 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T16:36:17.304 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:36:17.889 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T16:36:18.549 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-25T16:36:19.323 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-25T16:36:19.751 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-03-25T16:36:22.380 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.0 MB 00:02 2024-03-25T16:36:24.731 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-25T16:36:30.757 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.1 MB 00:02 2024-03-25T16:36:33.258 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T16:36:36.855 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-03-25T16:36:41.023 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-25T16:36:42.599 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-25T16:36:44.053 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T16:36:48.961 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T16:36:51.394 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:36:51.417 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrading: 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout: boost-program-options 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:36:51.418 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:36:51.419 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout:Installing weak dependencies: 2024-03-25T16:36:51.420 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:36:51.421 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:36:51.421 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:36:51.421 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:36:51.421 INFO:teuthology.orchestra.run.smithi133.stdout:Install 30 Packages 2024-03-25T16:36:51.421 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrade 2 Packages 2024-03-25T16:36:51.421 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:36:51.421 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 58 M 2024-03-25T16:36:51.421 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:36:54.386 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:21 2024-03-25T16:36:59.416 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T16:37:01.955 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:37:01.971 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:37:01.972 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:37:01.973 INFO:teuthology.orchestra.run.smithi055.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:37:01.974 INFO:teuthology.orchestra.run.smithi055.stdout:Installing weak dependencies: 2024-03-25T16:37:01.974 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:37:01.974 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:37:01.974 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:37:01.974 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:37:01.974 INFO:teuthology.orchestra.run.smithi055.stdout:Install 30 Packages 2024-03-25T16:37:01.974 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 2 Packages 2024-03-25T16:37:01.974 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:37:01.975 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 58 M 2024-03-25T16:37:01.975 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:37:07.482 INFO:teuthology.orchestra.run.smithi133.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 346 kB/s | 5.3 MB 00:15 2024-03-25T16:37:07.608 INFO:teuthology.orchestra.run.smithi133.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 202 kB/s | 25 kB 00:00 2024-03-25T16:37:11.293 INFO:teuthology.orchestra.run.smithi133.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 201 kB/s | 742 kB 00:03 2024-03-25T16:37:13.885 INFO:teuthology.orchestra.run.smithi133.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 192 kB/s | 498 kB 00:02 2024-03-25T16:37:25.483 INFO:teuthology.orchestra.run.smithi055.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 237 kB/s | 5.3 MB 00:23 2024-03-25T16:37:25.591 INFO:teuthology.orchestra.run.smithi055.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 233 kB/s | 25 kB 00:00 2024-03-25T16:37:28.133 INFO:teuthology.orchestra.run.smithi055.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 292 kB/s | 742 kB 00:02 2024-03-25T16:37:28.866 INFO:teuthology.orchestra.run.smithi133.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 349 kB/s | 5.1 MB 00:14 2024-03-25T16:37:28.991 INFO:teuthology.orchestra.run.smithi133.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 362 kB/s | 45 kB 00:00 2024-03-25T16:37:29.233 INFO:teuthology.orchestra.run.smithi055.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 372 kB/s | 9.7 MB 00:26 2024-03-25T16:37:29.308 INFO:teuthology.orchestra.run.smithi133.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 415 kB/s | 131 kB 00:00 2024-03-25T16:37:29.933 INFO:teuthology.orchestra.run.smithi133.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 259 kB/s | 162 kB 00:00 2024-03-25T16:37:30.375 INFO:teuthology.orchestra.run.smithi055.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 222 kB/s | 498 kB 00:02 2024-03-25T16:37:30.717 INFO:teuthology.orchestra.run.smithi133.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 409 kB/s | 320 kB 00:00 2024-03-25T16:37:30.842 INFO:teuthology.orchestra.run.smithi055.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 97 kB/s | 45 kB 00:00 2024-03-25T16:37:31.492 INFO:teuthology.orchestra.run.smithi133.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 384 kB/s | 298 kB 00:00 2024-03-25T16:37:31.809 INFO:teuthology.orchestra.run.smithi133.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 314 kB/s | 99 kB 00:00 2024-03-25T16:37:31.917 INFO:teuthology.orchestra.run.smithi133.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T16:37:31.960 INFO:teuthology.orchestra.run.smithi133.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:37:31.984 INFO:teuthology.orchestra.run.smithi055.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 115 kB/s | 131 kB 00:01 2024-03-25T16:37:31.993 INFO:teuthology.orchestra.run.smithi133.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:37:32.052 INFO:teuthology.orchestra.run.smithi133.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:37:32.094 INFO:teuthology.orchestra.run.smithi133.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:37:32.127 INFO:teuthology.orchestra.run.smithi133.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:37:32.261 INFO:teuthology.orchestra.run.smithi133.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.6 MB/s | 216 kB 00:00 2024-03-25T16:37:32.303 INFO:teuthology.orchestra.run.smithi133.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 839 kB/s | 35 kB 00:00 2024-03-25T16:37:32.345 INFO:teuthology.orchestra.run.smithi133.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:37:32.453 INFO:teuthology.orchestra.run.smithi133.stdout:(21/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 245 kB/s | 9.7 MB 00:40 2024-03-25T16:37:32.887 INFO:teuthology.orchestra.run.smithi133.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 570 kB/s | 309 kB 00:00 2024-03-25T16:37:32.970 INFO:teuthology.orchestra.run.smithi133.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-03-25T16:37:33.062 INFO:teuthology.orchestra.run.smithi133.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 531 kB/s | 49 kB 00:00 2024-03-25T16:37:33.163 INFO:teuthology.orchestra.run.smithi133.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 675 kB/s | 67 kB 00:00 2024-03-25T16:37:33.230 INFO:teuthology.orchestra.run.smithi133.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-03-25T16:37:33.305 INFO:teuthology.orchestra.run.smithi133.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T16:37:33.355 INFO:teuthology.orchestra.run.smithi133.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-25T16:37:33.414 INFO:teuthology.orchestra.run.smithi133.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-25T16:37:33.468 INFO:teuthology.orchestra.run.smithi055.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 109 kB/s | 162 kB 00:01 2024-03-25T16:37:35.693 INFO:teuthology.orchestra.run.smithi055.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 144 kB/s | 320 kB 00:02 2024-03-25T16:37:37.360 INFO:teuthology.orchestra.run.smithi055.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 179 kB/s | 298 kB 00:01 2024-03-25T16:37:38.327 INFO:teuthology.orchestra.run.smithi055.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 103 kB/s | 99 kB 00:00 2024-03-25T16:37:38.435 INFO:teuthology.orchestra.run.smithi055.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 982 kB/s | 106 kB 00:00 2024-03-25T16:37:38.477 INFO:teuthology.orchestra.run.smithi055.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:37:38.510 INFO:teuthology.orchestra.run.smithi055.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:37:38.561 INFO:teuthology.orchestra.run.smithi055.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-25T16:37:38.603 INFO:teuthology.orchestra.run.smithi055.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:37:38.636 INFO:teuthology.orchestra.run.smithi055.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:37:38.862 INFO:teuthology.orchestra.run.smithi055.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 963 kB/s | 216 kB 00:00 2024-03-25T16:37:38.921 INFO:teuthology.orchestra.run.smithi055.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 598 kB/s | 35 kB 00:00 2024-03-25T16:37:38.979 INFO:teuthology.orchestra.run.smithi055.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-25T16:37:39.071 INFO:teuthology.orchestra.run.smithi055.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.3 MB/s | 309 kB 00:00 2024-03-25T16:37:39.213 INFO:teuthology.orchestra.run.smithi055.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-03-25T16:37:39.247 INFO:teuthology.orchestra.run.smithi055.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 782 kB/s | 26 kB 00:00 2024-03-25T16:37:39.280 INFO:teuthology.orchestra.run.smithi055.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-25T16:37:39.314 INFO:teuthology.orchestra.run.smithi055.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-25T16:37:39.364 INFO:teuthology.orchestra.run.smithi055.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2024-03-25T16:37:39.398 INFO:teuthology.orchestra.run.smithi055.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-03-25T16:37:39.457 INFO:teuthology.orchestra.run.smithi055.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-03-25T16:37:42.647 INFO:teuthology.orchestra.run.smithi133.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 347 kB/s | 3.1 MB 00:09 2024-03-25T16:37:49.248 INFO:teuthology.orchestra.run.smithi055.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 261 kB/s | 5.1 MB 00:20 2024-03-25T16:37:49.495 INFO:teuthology.orchestra.run.smithi133.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 214 kB/s | 3.4 MB 00:16 2024-03-25T16:37:55.439 INFO:teuthology.orchestra.run.smithi055.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 216 kB/s | 3.4 MB 00:15 2024-03-25T16:37:58.214 INFO:teuthology.orchestra.run.smithi055.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 357 kB/s | 3.1 MB 00:08 2024-03-25T16:37:59.044 INFO:teuthology.orchestra.run.smithi133.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 302 kB/s | 20 MB 01:07 2024-03-25T16:37:59.050 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:59.050 INFO:teuthology.orchestra.run.smithi133.stdout:Total 876 kB/s | 58 MB 01:07 2024-03-25T16:37:59.083 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:37:59.121 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:37:59.122 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:37:59.599 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:37:59.599 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:38:01.068 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:38:01.168 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:38:01.207 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:38:01.232 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:38:01.255 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:38:01.271 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:38:01.463 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:38:01.535 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:38:01.854 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:38:01.934 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:38:02.025 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:38:02.112 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:38:02.135 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:38:02.181 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:38:02.210 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:38:02.253 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:38:02.321 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:38:02.345 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:38:02.408 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:38:02.435 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:38:02.543 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:38:02.579 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:38:02.627 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:38:02.660 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:38:02.688 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:38:02.721 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:38:03.063 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:38:03.093 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:38:03.183 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:38:03.215 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:38:03.256 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:38:03.729 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:38:03.754 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:38:03.825 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:38:03.894 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:38:03.923 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:38:06.110 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:38:06.161 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:38:06.708 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:38:06.747 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:38:06.821 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:38:06.821 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:38:06.821 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:38:06.822 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:38:06.865 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:38:15.988 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:38:15.988 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /sys 2024-03-25T16:38:15.988 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /proc 2024-03-25T16:38:15.988 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /mnt 2024-03-25T16:38:15.988 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /var/tmp 2024-03-25T16:38:15.988 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /home 2024-03-25T16:38:15.988 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /root 2024-03-25T16:38:15.988 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /tmp 2024-03-25T16:38:15.989 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:38:16.884 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:38:16.909 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:38:16.909 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:16.909 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:38:16.909 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:38:16.909 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:38:16.909 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:38:16.912 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:38:16.996 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:38:16.996 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:38:18.080 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:38:18.080 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:38:18.081 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:38:18.082 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:38:18.083 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:38:18.083 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:38:18.083 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:38:18.083 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:38:19.293 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:38:19.293 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout:Upgraded: 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:38:19.294 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.295 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:38:19.296 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:38:19.538 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-test 2024-03-25T16:38:20.094 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:32 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:38:20.252 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:38:20.253 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:38:20.254 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:38:20.254 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:38:20.254 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2024-03-25T16:38:20.254 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:38:20.254 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 48 M 2024-03-25T16:38:20.254 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 204 M 2024-03-25T16:38:20.254 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:38:20.608 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 699 kB/s | 217 kB 00:00 2024-03-25T16:38:20.733 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-03-25T16:38:20.800 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 940 kB/s | 63 kB 00:00 2024-03-25T16:38:21.275 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 394 kB/s | 187 kB 00:00 2024-03-25T16:38:21.508 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 135 kB/s | 163 kB 00:01 2024-03-25T16:38:27.120 INFO:teuthology.orchestra.run.smithi055.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 240 kB/s | 20 MB 01:24 2024-03-25T16:38:27.124 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:27.124 INFO:teuthology.orchestra.run.smithi055.stdout:Total 695 kB/s | 58 MB 01:25 2024-03-25T16:38:27.154 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:38:27.194 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:38:27.194 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:38:27.680 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:38:27.680 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:38:29.166 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:38:29.315 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:38:29.350 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:38:29.375 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:38:29.406 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:38:29.422 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:38:29.615 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:38:29.671 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:38:30.081 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:38:30.111 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:38:30.195 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:38:30.285 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:38:30.313 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:38:30.361 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:38:30.402 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:38:30.441 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:38:30.518 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:38:30.557 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:38:30.620 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:38:30.648 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:38:30.759 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:38:30.793 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:38:30.831 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:38:30.864 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:38:30.893 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:38:30.926 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:38:31.262 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:38:31.289 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:38:31.363 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:38:31.394 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:38:31.440 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:38:31.918 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:38:31.950 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:38:32.005 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:38:32.076 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:38:32.120 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:38:34.357 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:38:34.392 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:38:34.984 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:38:35.029 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:38:35.116 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:38:35.116 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:38:35.116 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:38:35.116 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:38:35.180 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:38:44.824 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:38:44.824 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2024-03-25T16:38:44.824 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2024-03-25T16:38:44.824 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2024-03-25T16:38:44.824 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2024-03-25T16:38:44.824 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2024-03-25T16:38:44.825 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /root 2024-03-25T16:38:44.825 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2024-03-25T16:38:44.825 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:38:45.755 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:38:45.779 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:38:45.779 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:45.779 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:38:45.779 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:38:45.779 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:38:45.779 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:38:45.781 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:38:45.864 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:38:45.865 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:38:46.971 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:38:46.971 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:38:46.971 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:38:46.971 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:38:46.971 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:38:46.972 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:38:46.974 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:38:46.974 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:38:48.029 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:38:48.029 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:38:48.029 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2024-03-25T16:38:48.029 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:38:48.030 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:38:48.031 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:38:48.032 INFO:teuthology.orchestra.run.smithi055.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:38:48.032 INFO:teuthology.orchestra.run.smithi055.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:38:48.032 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:38:48.032 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:38:48.357 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-test 2024-03-25T16:38:48.928 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:49 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:38:49.094 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:38:49.095 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:38:49.095 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:38:49.095 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:38:49.095 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:38:49.095 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2024-03-25T16:38:49.096 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:38:49.097 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 48 M 2024-03-25T16:38:49.097 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 204 M 2024-03-25T16:38:49.097 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:38:49.277 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-25T16:38:49.352 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-25T16:38:49.385 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:38:49.889 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 374 kB/s | 187 kB 00:00 2024-03-25T16:38:49.928 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 208 kB/s | 163 kB 00:00 2024-03-25T16:41:23.008 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 265 kB/s | 47 MB 03:02 2024-03-25T16:41:23.009 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:41:23.009 INFO:teuthology.orchestra.run.smithi133.stdout:Total 270 kB/s | 48 MB 03:02 2024-03-25T16:41:23.021 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:41:23.040 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:41:23.040 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:41:23.417 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:41:23.417 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:41:24.016 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:41:24.081 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:41:24.118 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:41:24.149 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:41:24.199 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:41:24.218 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:41:28.669 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:41:28.742 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:41:31.164 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:41:31.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:41:31.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:41:31.164 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:41:31.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:41:31.165 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:41:31.559 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:41:31.559 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:41:31.559 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:41:31.560 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:31.560 INFO:teuthology.orchestra.run.smithi133.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:41:31.560 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:31.560 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:41:31.560 INFO:teuthology.orchestra.run.smithi133.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:41:31.560 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:41:31.560 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:41:31.560 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:41:31.877 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph 2024-03-25T16:41:32.443 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:41:32.596 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:41:32.599 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================= 2024-03-25T16:41:32.599 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:41:32.599 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================= 2024-03-25T16:41:32.599 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:41:32.599 INFO:teuthology.orchestra.run.smithi133.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:41:32.600 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:41:32.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout:========================================================================================= 2024-03-25T16:41:32.602 INFO:teuthology.orchestra.run.smithi133.stdout:Install 36 Packages 2024-03-25T16:41:32.603 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:41:32.603 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 30 M 2024-03-25T16:41:32.603 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 107 M 2024-03-25T16:41:32.603 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:41:33.226 INFO:teuthology.orchestra.run.smithi133.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:41:39.259 INFO:teuthology.orchestra.run.smithi133.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 235 kB/s | 1.5 MB 00:06 2024-03-25T16:41:43.291 INFO:teuthology.orchestra.run.smithi133.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 207 kB/s | 2.1 MB 00:10 2024-03-25T16:41:43.893 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 277 kB/s | 47 MB 02:54 2024-03-25T16:41:43.895 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:41:43.895 INFO:teuthology.orchestra.run.smithi055.stdout:Total 282 kB/s | 48 MB 02:54 2024-03-25T16:41:43.907 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:41:43.927 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:41:43.927 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:41:44.320 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:41:44.321 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:41:44.924 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:41:45.002 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:41:45.045 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:41:45.078 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:41:45.123 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:41:45.146 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:41:46.358 INFO:teuthology.orchestra.run.smithi133.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 81 kB/s | 247 kB 00:03 2024-03-25T16:41:46.500 INFO:teuthology.orchestra.run.smithi133.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T16:41:46.541 INFO:teuthology.orchestra.run.smithi133.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:41:46.575 INFO:teuthology.orchestra.run.smithi133.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-25T16:41:46.609 INFO:teuthology.orchestra.run.smithi133.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:41:46.642 INFO:teuthology.orchestra.run.smithi133.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:41:46.693 INFO:teuthology.orchestra.run.smithi133.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:41:46.760 INFO:teuthology.orchestra.run.smithi133.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:41:46.793 INFO:teuthology.orchestra.run.smithi133.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:41:46.827 INFO:teuthology.orchestra.run.smithi133.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:41:46.869 INFO:teuthology.orchestra.run.smithi133.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:41:47.011 INFO:teuthology.orchestra.run.smithi133.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 881 kB/s | 125 kB 00:00 2024-03-25T16:41:47.145 INFO:teuthology.orchestra.run.smithi133.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-03-25T16:41:47.520 INFO:teuthology.orchestra.run.smithi133.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 403 kB/s | 151 kB 00:00 2024-03-25T16:41:47.620 INFO:teuthology.orchestra.run.smithi133.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 434 kB/s | 43 kB 00:00 2024-03-25T16:41:47.820 INFO:teuthology.orchestra.run.smithi133.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 859 kB/s | 172 kB 00:00 2024-03-25T16:41:48.062 INFO:teuthology.orchestra.run.smithi133.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-25T16:41:48.154 INFO:teuthology.orchestra.run.smithi133.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-03-25T16:41:48.238 INFO:teuthology.orchestra.run.smithi133.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-03-25T16:41:48.321 INFO:teuthology.orchestra.run.smithi133.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-03-25T16:41:48.405 INFO:teuthology.orchestra.run.smithi133.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-03-25T16:41:48.497 INFO:teuthology.orchestra.run.smithi133.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 215 kB/s | 20 kB 00:00 2024-03-25T16:41:48.580 INFO:teuthology.orchestra.run.smithi133.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 557 kB/s | 46 kB 00:00 2024-03-25T16:41:48.681 INFO:teuthology.orchestra.run.smithi133.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 789 kB/s | 79 kB 00:00 2024-03-25T16:41:48.781 INFO:teuthology.orchestra.run.smithi133.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T16:41:48.881 INFO:teuthology.orchestra.run.smithi133.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-25T16:41:48.981 INFO:teuthology.orchestra.run.smithi133.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 901 kB/s | 90 kB 00:00 2024-03-25T16:41:49.107 INFO:teuthology.orchestra.run.smithi133.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 286 kB/s | 36 kB 00:00 2024-03-25T16:41:49.215 INFO:teuthology.orchestra.run.smithi133.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-25T16:41:49.324 INFO:teuthology.orchestra.run.smithi133.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.9 MB/s | 427 kB 00:00 2024-03-25T16:41:49.424 INFO:teuthology.orchestra.run.smithi133.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-25T16:41:49.862 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:41:49.948 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:41:52.366 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:41:52.366 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:41:52.366 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:41:52.366 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:41:52.367 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:41:52.367 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:41:52.778 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:41:52.778 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:41:52.778 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:41:52.778 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:52.778 INFO:teuthology.orchestra.run.smithi055.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:41:52.778 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:52.778 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:41:52.779 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:41:52.779 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:41:52.779 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:41:52.779 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:41:53.088 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph 2024-03-25T16:41:53.655 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:04:54 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:41:53.816 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:41:53.819 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2024-03-25T16:41:53.819 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:41:53.819 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2024-03-25T16:41:53.819 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:41:53.819 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:41:53.819 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:41:53.819 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:41:53.819 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:41:53.819 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:41:53.820 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:41:53.821 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:41:53.822 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:41:53.822 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:41:53.822 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:41:53.822 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:41:53.822 INFO:teuthology.orchestra.run.smithi055.stdout:========================================================================================= 2024-03-25T16:41:53.822 INFO:teuthology.orchestra.run.smithi055.stdout:Install 36 Packages 2024-03-25T16:41:53.822 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:41:53.823 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 30 M 2024-03-25T16:41:53.823 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 107 M 2024-03-25T16:41:53.823 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:41:54.563 INFO:teuthology.orchestra.run.smithi055.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T16:41:55.399 INFO:teuthology.orchestra.run.smithi133.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 216 kB/s | 4.7 MB 00:22 2024-03-25T16:41:57.446 INFO:teuthology.orchestra.run.smithi055.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 467 kB/s | 1.5 MB 00:03 2024-03-25T16:42:03.063 INFO:teuthology.orchestra.run.smithi055.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 245 kB/s | 2.1 MB 00:08 2024-03-25T16:42:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 171 kB/s | 247 kB 00:01 2024-03-25T16:42:04.654 INFO:teuthology.orchestra.run.smithi055.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T16:42:04.713 INFO:teuthology.orchestra.run.smithi055.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T16:42:04.738 INFO:teuthology.orchestra.run.smithi055.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:42:04.772 INFO:teuthology.orchestra.run.smithi055.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:42:04.797 INFO:teuthology.orchestra.run.smithi055.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:42:04.848 INFO:teuthology.orchestra.run.smithi055.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:42:04.931 INFO:teuthology.orchestra.run.smithi055.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T16:42:04.973 INFO:teuthology.orchestra.run.smithi055.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:42:05.007 INFO:teuthology.orchestra.run.smithi055.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:42:05.048 INFO:teuthology.orchestra.run.smithi055.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:42:05.482 INFO:teuthology.orchestra.run.smithi055.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 288 kB/s | 125 kB 00:00 2024-03-25T16:42:05.599 INFO:teuthology.orchestra.run.smithi055.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-03-25T16:42:05.958 INFO:teuthology.orchestra.run.smithi055.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 422 kB/s | 151 kB 00:00 2024-03-25T16:42:06.075 INFO:teuthology.orchestra.run.smithi055.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 373 kB/s | 43 kB 00:00 2024-03-25T16:42:06.217 INFO:teuthology.orchestra.run.smithi055.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-03-25T16:42:06.417 INFO:teuthology.orchestra.run.smithi055.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T16:42:06.484 INFO:teuthology.orchestra.run.smithi055.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-25T16:42:06.560 INFO:teuthology.orchestra.run.smithi055.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-25T16:42:06.626 INFO:teuthology.orchestra.run.smithi055.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-25T16:42:06.710 INFO:teuthology.orchestra.run.smithi055.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-03-25T16:42:06.802 INFO:teuthology.orchestra.run.smithi055.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-25T16:42:06.902 INFO:teuthology.orchestra.run.smithi055.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-25T16:42:07.011 INFO:teuthology.orchestra.run.smithi055.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 727 kB/s | 79 kB 00:00 2024-03-25T16:42:07.128 INFO:teuthology.orchestra.run.smithi055.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-25T16:42:07.195 INFO:teuthology.orchestra.run.smithi055.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-03-25T16:42:07.270 INFO:teuthology.orchestra.run.smithi055.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T16:42:07.346 INFO:teuthology.orchestra.run.smithi055.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 476 kB/s | 36 kB 00:00 2024-03-25T16:42:07.446 INFO:teuthology.orchestra.run.smithi055.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-25T16:42:07.579 INFO:teuthology.orchestra.run.smithi055.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-25T16:42:07.688 INFO:teuthology.orchestra.run.smithi055.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-25T16:42:08.130 INFO:teuthology.orchestra.run.smithi055.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 354 kB/s | 4.7 MB 00:13 2024-03-25T16:42:43.577 INFO:teuthology.orchestra.run.smithi055.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 372 kB/s | 17 MB 00:46 2024-03-25T16:42:43.581 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:42:43.581 INFO:teuthology.orchestra.run.smithi055.stdout:Total 610 kB/s | 30 MB 00:49 2024-03-25T16:42:43.604 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:42:43.650 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:42:43.650 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:42:44.066 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:42:44.066 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:42:45.407 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:42:45.471 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:42:45.510 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:42:45.549 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:42:45.623 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:42:45.673 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:42:45.705 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:42:45.801 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:42:45.886 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:42:45.934 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:42:46.008 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:42:46.059 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:42:46.097 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:42:46.147 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:42:46.202 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:42:46.396 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:42:46.443 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:42:46.482 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:42:46.516 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:42:46.546 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:42:46.617 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:42:46.648 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:42:46.681 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:42:46.722 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:42:46.757 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:42:46.791 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:42:46.884 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:42:46.930 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:42:47.006 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:42:47.155 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:42:47.280 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:42:47.446 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:42:47.468 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:42:47.493 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:42:47.493 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:42:47.493 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:42:47.493 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:42:47.494 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:42:47.494 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:49.011 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:42:49.037 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:42:49.037 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:42:49.037 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:42:49.037 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:42:49.037 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:42:49.037 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:49.503 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:42:49.526 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:42:49.526 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:42:49.526 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:42:49.526 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:42:49.526 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:42:49.526 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:49.746 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:42:49.769 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:42:49.769 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:42:49.769 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:42:49.769 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:42:49.769 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:42:49.769 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:49.837 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:42:51.529 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:42:51.529 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:42:51.529 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:42:51.529 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:42:51.529 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:42:51.529 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:42:51.529 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:42:51.529 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:42:51.530 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:42:51.531 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:42:51.531 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:42:51.531 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:42:51.531 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:42:51.531 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:42:51.532 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:42:51.532 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:42:51.532 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:42:51.532 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:42:51.532 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:42:51.532 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:42:51.533 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:42:51.533 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:42:51.533 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:42:51.533 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:42:52.299 INFO:teuthology.orchestra.run.smithi133.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 235 kB/s | 17 MB 01:13 2024-03-25T16:42:52.302 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:42:52.303 INFO:teuthology.orchestra.run.smithi133.stdout:Total 381 kB/s | 30 MB 01:19 2024-03-25T16:42:52.323 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:42:52.364 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:42:52.364 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:42:52.584 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:42:52.585 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:42:52.586 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:52.587 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:42:52.770 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:42:52.770 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:42:52.962 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-base 2024-03-25T16:42:53.537 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:05:54 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:42:53.638 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:42:53.693 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:42:53.694 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T16:42:53.694 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:42:53.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install cephadm 2024-03-25T16:42:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:42:54.159 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:42:54.203 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:42:54.239 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:42:54.308 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:42:54.333 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:42:54.354 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:42:54.387 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:42:54.477 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:42:54.488 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T16:42:54.489 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:54.490 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 747 k 2024-03-25T16:42:54.490 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 754 k 2024-03-25T16:42:54.490 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:42:54.556 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:42:54.597 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:42:54.668 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:42:54.723 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:42:54.762 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:42:54.810 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:42:54.865 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:42:55.049 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:42:55.090 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:42:55.121 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:42:55.156 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:42:55.186 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:42:55.262 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:42:55.304 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:42:55.338 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:42:55.378 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:42:55.413 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:42:55.448 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:42:55.538 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:42:55.587 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:42:55.659 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:42:55.796 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:42:55.912 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:42:56.069 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:42:56.100 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:42:56.124 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:42:56.124 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:42:56.124 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:42:56.124 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:42:56.124 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:42:56.124 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:42:57.179 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 278 kB/s | 747 kB 00:02 2024-03-25T16:42:57.179 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:42:57.180 INFO:teuthology.orchestra.run.smithi055.stdout:Total 278 kB/s | 747 kB 00:02 2024-03-25T16:42:57.180 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:42:57.187 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:42:57.187 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:42:57.193 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:42:57.193 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:42:57.269 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:42:57.559 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:42:57.584 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:42:57.584 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:42:57.585 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:42:57.585 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:42:57.585 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:42:57.585 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:42:57.666 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:42:57.754 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:42:58.027 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:42:58.049 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:42:58.050 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:42:58.050 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:42:58.050 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:42:58.050 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:42:58.050 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:42:58.129 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:42:58.260 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:42:58.282 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:42:58.282 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:42:58.282 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:42:58.282 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:42:58.282 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:42:58.282 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:42:58.350 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:42:58.394 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:42:58.394 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:58.394 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:42:58.394 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:42:58.394 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:58.394 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:42:58.593 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:42:59.166 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:42:59.322 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:42:59.322 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:42:59.322 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T16:42:59.323 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:42:59.323 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:42:59.323 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:42:59.323 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:59.323 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:42:59.323 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:42:59.323 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T16:42:59.323 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:42:59.324 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 145 k 2024-03-25T16:42:59.324 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 436 k 2024-03-25T16:42:59.324 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:42:59.996 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:42:59.996 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:42:59.996 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:42:59.996 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:42:59.996 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:42:59.996 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:42:59.996 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:42:59.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:42:59.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:42:59.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:42:59.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:42:59.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:42:59.999 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:42:59.999 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:42:59.999 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:42:59.999 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:42:59.999 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:42:59.999 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:42:59.999 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:42:59.999 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:43:00.118 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 183 kB/s | 145 kB 00:00 2024-03-25T16:43:00.119 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:00.119 INFO:teuthology.orchestra.run.smithi055.stdout:Total 183 kB/s | 145 kB 00:00 2024-03-25T16:43:00.120 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:43:00.128 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:43:00.128 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:43:00.184 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:43:00.184 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:43:00.325 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:43:00.394 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:43:00.418 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:43:00.418 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:43:00.418 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:43:00.418 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:43:01.043 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:43:01.044 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:43:01.045 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:01.046 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:43:01.364 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base 2024-03-25T16:43:01.367 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:43:01.367 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:43:01.367 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:43:01.367 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:01.367 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:43:01.367 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:43:01.609 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr 2024-03-25T16:43:01.937 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:06:13 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:43:02.035 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:43:02.087 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:43:02.088 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T16:43:02.088 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:43:02.156 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install cephadm 2024-03-25T16:43:02.182 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:43:02.281 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:43:02.338 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:43:02.338 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T16:43:02.338 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:43:02.399 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:43:02.716 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:43:02.867 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:43:02.868 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:43:02.868 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:43:02.868 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:43:02.868 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:43:02.868 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:43:02.868 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:02.868 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:43:02.868 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:43:02.868 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T16:43:02.869 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:02.869 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 747 k 2024-03-25T16:43:02.869 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 754 k 2024-03-25T16:43:02.869 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:43:02.970 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:43:03.125 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:43:03.126 INFO:teuthology.orchestra.run.smithi055.stdout:====================================================================================== 2024-03-25T16:43:03.126 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:43:03.126 INFO:teuthology.orchestra.run.smithi055.stdout:====================================================================================== 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:43:03.127 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:43:03.128 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:43:03.128 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:43:03.128 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:43:03.128 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:43:03.128 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:43:03.128 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:43:03.128 INFO:teuthology.orchestra.run.smithi055.stdout:====================================================================================== 2024-03-25T16:43:03.128 INFO:teuthology.orchestra.run.smithi055.stdout:Install 12 Packages 2024-03-25T16:43:03.128 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:43:03.129 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 6.9 M 2024-03-25T16:43:03.129 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 26 M 2024-03-25T16:43:03.129 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:43:03.797 INFO:teuthology.orchestra.run.smithi055.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T16:43:03.947 INFO:teuthology.orchestra.run.smithi055.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-25T16:43:03.972 INFO:teuthology.orchestra.run.smithi055.stdout:(3/12): ceph-grafana-dashboards-19.0.0-1439.g5d 48 kB/s | 31 kB 00:00 2024-03-25T16:43:03.997 INFO:teuthology.orchestra.run.smithi055.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 4.6 MB/s | 238 kB 00:00 2024-03-25T16:43:04.206 INFO:teuthology.orchestra.run.smithi055.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-03-25T16:43:04.256 INFO:teuthology.orchestra.run.smithi055.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-25T16:43:04.281 INFO:teuthology.orchestra.run.smithi055.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 261 kB/s | 19 kB 00:00 2024-03-25T16:43:04.390 INFO:teuthology.orchestra.run.smithi055.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-03-25T16:43:04.415 INFO:teuthology.orchestra.run.smithi055.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-25T16:43:04.441 INFO:teuthology.orchestra.run.smithi055.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 614 kB/s | 31 kB 00:00 2024-03-25T16:43:04.466 INFO:teuthology.orchestra.run.smithi055.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-25T16:43:08.448 INFO:teuthology.orchestra.run.smithi133.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 134 kB/s | 747 kB 00:05 2024-03-25T16:43:08.449 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:08.449 INFO:teuthology.orchestra.run.smithi133.stdout:Total 134 kB/s | 747 kB 00:05 2024-03-25T16:43:08.449 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:43:08.456 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:43:08.456 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:43:08.462 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:43:08.462 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:43:08.539 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:43:08.975 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:43:09.070 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:43:09.457 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:43:09.745 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:43:09.745 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:09.745 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:43:09.746 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:43:09.746 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:09.746 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:43:09.935 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:43:10.507 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:06:22 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:43:10.625 INFO:teuthology.orchestra.run.smithi055.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 250 kB/s | 1.8 MB 00:07 2024-03-25T16:43:10.628 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:10.629 INFO:teuthology.orchestra.run.smithi055.stdout:Total 936 kB/s | 6.9 MB 00:07 2024-03-25T16:43:10.649 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:43:10.663 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T16:43:10.664 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:10.665 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 145 k 2024-03-25T16:43:10.665 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 436 k 2024-03-25T16:43:10.665 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:43:10.669 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:43:10.669 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:43:10.860 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:43:10.860 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:43:11.522 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:43:11.600 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:43:11.638 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 149 kB/s | 145 kB 00:00 2024-03-25T16:43:11.638 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:11.639 INFO:teuthology.orchestra.run.smithi133.stdout:Total 149 kB/s | 145 kB 00:00 2024-03-25T16:43:11.639 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:43:11.646 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:43:11.647 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:43:11.652 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:43:11.699 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:43:11.700 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:43:11.700 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:43:11.737 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:43:11.836 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:43:11.842 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:43:11.913 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:43:11.938 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:43:11.939 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:43:11.939 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:43:11.939 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:11.967 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:43:12.068 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:43:12.117 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:43:12.153 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:43:12.186 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:43:12.442 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:43:12.476 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:43:12.925 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:43:12.925 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:12.925 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:43:12.925 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:12.925 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:12.925 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:43:13.149 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr 2024-03-25T16:43:13.215 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:43:13.215 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:43:13.215 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:43:13.215 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:43:13.215 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:43:13.216 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:43:13.216 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:43:13.216 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:43:13.216 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:43:13.216 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:43:13.216 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:43:13.216 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:43:13.561 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:43:13.561 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:43:13.561 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:43:13.561 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:43:13.561 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:43:13.561 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:43:13.562 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:43:13.719 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:06:25 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:43:13.810 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:43:13.818 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:43:13.870 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:43:13.871 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T16:43:13.871 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:43:13.938 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:43:14.381 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:43:14.498 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:43:14.536 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:43:14.537 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:43:14.538 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:43:14.539 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:43:14.539 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:43:14.539 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:43:14.539 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:43:14.539 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:43:14.539 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================================ 2024-03-25T16:43:14.539 INFO:teuthology.orchestra.run.smithi055.stdout:Install 12 Packages 2024-03-25T16:43:14.539 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:43:14.540 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 37 M 2024-03-25T16:43:14.540 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 209 M 2024-03-25T16:43:14.540 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:43:14.647 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:43:14.648 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2024-03-25T16:43:14.648 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:43:14.648 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2024-03-25T16:43:14.648 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:43:14.648 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:43:14.648 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:43:14.649 INFO:teuthology.orchestra.run.smithi133.stdout:====================================================================================== 2024-03-25T16:43:14.650 INFO:teuthology.orchestra.run.smithi133.stdout:Install 12 Packages 2024-03-25T16:43:14.650 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:14.650 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 6.9 M 2024-03-25T16:43:14.650 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 26 M 2024-03-25T16:43:14.650 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:43:15.022 INFO:teuthology.orchestra.run.smithi055.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 143 kB/s | 32 kB 00:00 2024-03-25T16:43:15.064 INFO:teuthology.orchestra.run.smithi055.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-03-25T16:43:15.106 INFO:teuthology.orchestra.run.smithi055.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2024-03-25T16:43:15.239 INFO:teuthology.orchestra.run.smithi055.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.8 MB/s | 3.0 MB 00:00 2024-03-25T16:43:15.473 INFO:teuthology.orchestra.run.smithi055.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-03-25T16:43:15.540 INFO:teuthology.orchestra.run.smithi055.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-25T16:43:15.565 INFO:teuthology.orchestra.run.smithi055.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:43:15.774 INFO:teuthology.orchestra.run.smithi133.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 27 kB/s | 15 kB 00:00 2024-03-25T16:43:15.833 INFO:teuthology.orchestra.run.smithi133.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 49 kB/s | 31 kB 00:00 2024-03-25T16:43:15.908 INFO:teuthology.orchestra.run.smithi055.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-25T16:43:15.924 INFO:teuthology.orchestra.run.smithi133.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.6 MB/s | 238 kB 00:00 2024-03-25T16:43:15.958 INFO:teuthology.orchestra.run.smithi133.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-25T16:43:16.091 INFO:teuthology.orchestra.run.smithi055.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-25T16:43:16.141 INFO:teuthology.orchestra.run.smithi055.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 841 kB/s | 195 kB 00:00 2024-03-25T16:43:16.183 INFO:teuthology.orchestra.run.smithi133.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.4 MB/s | 548 kB 00:00 2024-03-25T16:43:16.242 INFO:teuthology.orchestra.run.smithi133.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-25T16:43:16.367 INFO:teuthology.orchestra.run.smithi133.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.9 MB/s | 863 kB 00:00 2024-03-25T16:43:16.435 INFO:teuthology.orchestra.run.smithi133.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-03-25T16:43:16.449 INFO:teuthology.orchestra.run.smithi055.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 685 kB/s | 245 kB 00:00 2024-03-25T16:43:16.468 INFO:teuthology.orchestra.run.smithi133.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-03-25T16:43:16.493 INFO:teuthology.orchestra.run.smithi133.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 528 kB/s | 31 kB 00:00 2024-03-25T16:43:16.519 INFO:teuthology.orchestra.run.smithi133.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-25T16:43:20.527 INFO:teuthology.orchestra.run.smithi133.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 342 kB/s | 1.8 MB 00:05 2024-03-25T16:43:20.531 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:20.532 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.2 MB/s | 6.9 MB 00:05 2024-03-25T16:43:20.552 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:43:20.571 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:43:20.571 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:43:20.762 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:43:20.763 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:43:21.278 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:43:21.352 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:43:21.387 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:43:21.434 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:43:21.464 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:43:21.564 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:43:21.698 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:43:21.792 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:43:21.835 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:43:21.863 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:43:21.895 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:43:22.138 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:43:22.179 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:43:22.997 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:43:22.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:43:22.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:43:22.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:43:22.997 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:43:22.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:43:22.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:43:22.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:43:22.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:43:22.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:43:22.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:43:22.998 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:43:23.349 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:43:23.349 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:43:23.350 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:23.351 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:43:23.645 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:43:24.208 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:06:36 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:43:24.360 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:43:24.361 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================================ 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================================ 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:43:24.362 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================================ 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout:Install 12 Packages 2024-03-25T16:43:24.363 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:24.364 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 37 M 2024-03-25T16:43:24.364 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 209 M 2024-03-25T16:43:24.364 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:43:24.624 INFO:teuthology.orchestra.run.smithi133.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 207 kB/s | 32 kB 00:00 2024-03-25T16:43:24.657 INFO:teuthology.orchestra.run.smithi133.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 515 kB/s | 17 kB 00:00 2024-03-25T16:43:24.682 INFO:teuthology.orchestra.run.smithi133.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T16:43:24.866 INFO:teuthology.orchestra.run.smithi133.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-25T16:43:25.017 INFO:teuthology.orchestra.run.smithi133.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-25T16:43:25.067 INFO:teuthology.orchestra.run.smithi133.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.3 MB/s | 217 kB 00:00 2024-03-25T16:43:25.434 INFO:teuthology.orchestra.run.smithi133.stdout:(7/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-25T16:43:25.492 INFO:teuthology.orchestra.run.smithi133.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-03-25T16:43:25.693 INFO:teuthology.orchestra.run.smithi133.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 977 kB/s | 195 kB 00:00 2024-03-25T16:43:25.851 INFO:teuthology.orchestra.run.smithi133.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 803 kB 00:00 2024-03-25T16:43:26.251 INFO:teuthology.orchestra.run.smithi133.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 439 kB/s | 245 kB 00:00 2024-03-25T16:43:46.840 INFO:teuthology.orchestra.run.smithi133.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 338 kB/s | 7.4 MB 00:22 2024-03-25T16:43:46.842 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:46.843 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.6 MB/s | 37 MB 00:22 2024-03-25T16:43:46.887 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:43:46.912 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:43:46.912 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:43:47.272 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:43:47.272 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:43:47.983 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:43:48.046 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:43:48.078 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:43:48.178 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:43:48.222 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:43:48.401 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:43:48.429 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:43:48.511 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:43:49.159 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:43:49.282 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:43:50.494 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:43:51.519 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:43:51.553 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:43:54.098 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:43:54.098 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:43:54.098 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:43:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:43:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:43:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:43:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:43:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:43:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:43:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:43:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:43:54.099 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:43:54.531 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:43:54.532 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:43:54.532 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:43:54.532 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:43:54.532 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:43:54.532 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:54.532 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:43:54.563 INFO:teuthology.orchestra.run.smithi055.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 190 kB/s | 7.4 MB 00:39 2024-03-25T16:43:54.567 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:54.567 INFO:teuthology.orchestra.run.smithi055.stdout:Total 948 kB/s | 37 MB 00:40 2024-03-25T16:43:54.608 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:43:54.635 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:43:54.635 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:43:54.907 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:43:55.010 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:43:55.011 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:43:55.478 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:07:07 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:43:55.627 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:43:55.629 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout:Install 13 Packages 2024-03-25T16:43:55.630 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:43:55.631 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 2.0 M 2024-03-25T16:43:55.631 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 26 M 2024-03-25T16:43:55.631 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:43:55.749 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:43:55.819 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:43:55.837 INFO:teuthology.orchestra.run.smithi133.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 159 kB/s | 18 kB 00:00 2024-03-25T16:43:55.850 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:43:55.879 INFO:teuthology.orchestra.run.smithi133.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 156 kB/s | 24 kB 00:00 2024-03-25T16:43:55.921 INFO:teuthology.orchestra.run.smithi133.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.8 MB/s | 174 kB 00:00 2024-03-25T16:43:55.946 INFO:teuthology.orchestra.run.smithi133.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T16:43:55.952 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:43:55.972 INFO:teuthology.orchestra.run.smithi133.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:43:55.987 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:43:55.988 INFO:teuthology.orchestra.run.smithi133.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T16:43:56.172 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:43:56.202 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:43:56.287 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:43:56.372 INFO:teuthology.orchestra.run.smithi133.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 80 kB/s | 52 kB 00:00 2024-03-25T16:43:56.656 INFO:teuthology.orchestra.run.smithi133.stdout:(8/13): python3-google-auth-2.28.2-1.el9.noarch 715 kB/s | 203 kB 00:00 2024-03-25T16:43:56.856 INFO:teuthology.orchestra.run.smithi133.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 5.1 MB/s | 1.0 MB 00:00 2024-03-25T16:43:56.931 INFO:teuthology.orchestra.run.smithi133.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 789 kB/s | 59 kB 00:00 2024-03-25T16:43:56.968 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:43:57.015 INFO:teuthology.orchestra.run.smithi133.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-25T16:43:57.077 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:43:58.392 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:43:59.469 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:43:59.502 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:44:02.351 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:44:02.351 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:44:02.351 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:44:02.351 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:44:02.351 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:44:02.351 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:44:02.351 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:44:02.352 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:44:02.352 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:44:02.352 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:44:02.352 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:44:02.352 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:44:02.774 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:44:02.774 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:44:02.775 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:02.776 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:03.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:44:03.656 INFO:teuthology.orchestra.run.smithi133.stdout:(12/13): python3-cachetools-4.2.4-1.el9.noarch. 4.2 kB/s | 32 kB 00:07 2024-03-25T16:44:03.807 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:04 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:03.963 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:03.964 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-25T16:44:03.964 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:44:03.964 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-25T16:44:03.964 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:03.964 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:44:03.964 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:44:03.965 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:03.966 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:03.966 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-25T16:44:03.966 INFO:teuthology.orchestra.run.smithi055.stdout:Install 13 Packages 2024-03-25T16:44:03.966 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:03.966 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.0 M 2024-03-25T16:44:03.966 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 26 M 2024-03-25T16:44:03.967 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:44:04.444 INFO:teuthology.orchestra.run.smithi055.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 210 kB/s | 24 kB 00:00 2024-03-25T16:44:04.494 INFO:teuthology.orchestra.run.smithi055.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-03-25T16:44:04.570 INFO:teuthology.orchestra.run.smithi055.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-25T16:44:04.595 INFO:teuthology.orchestra.run.smithi055.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-25T16:44:04.645 INFO:teuthology.orchestra.run.smithi055.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.8 MB/s | 211 kB 00:00 2024-03-25T16:44:04.670 INFO:teuthology.orchestra.run.smithi055.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 568 kB/s | 43 kB 00:00 2024-03-25T16:44:04.788 INFO:teuthology.orchestra.run.smithi055.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-25T16:44:04.804 INFO:teuthology.orchestra.run.smithi055.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 203 kB/s | 32 kB 00:00 2024-03-25T16:44:05.030 INFO:teuthology.orchestra.run.smithi055.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 838 kB/s | 203 kB 00:00 2024-03-25T16:44:05.046 INFO:teuthology.orchestra.run.smithi055.stdout:(10/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 73 kB/s | 52 kB 00:00 2024-03-25T16:44:05.072 INFO:teuthology.orchestra.run.smithi055.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-25T16:44:05.147 INFO:teuthology.orchestra.run.smithi055.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-25T16:44:05.214 INFO:teuthology.orchestra.run.smithi055.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 537 kB/s | 90 kB 00:00 2024-03-25T16:44:05.216 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:05.216 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-25T16:44:05.217 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:44:05.232 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:44:05.232 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:44:05.332 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:44:05.334 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:44:05.854 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:44:06.027 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:44:06.083 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:44:06.148 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:44:06.192 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:44:06.234 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:44:06.330 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:44:06.439 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:44:06.493 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:44:07.252 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:44:07.298 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:44:07.331 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:44:07.383 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:44:07.422 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:44:08.191 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:44:08.191 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:44:08.191 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:44:08.191 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:44:08.191 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:44:08.191 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:44:08.192 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:44:08.192 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:44:08.192 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:44:08.192 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:44:08.192 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:44:08.192 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:44:08.192 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:44:08.563 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:44:08.564 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:08.565 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:08.846 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:44:09.422 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:10 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:09.579 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:09.580 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-25T16:44:09.580 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:44:09.580 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-25T16:44:09.580 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2024-03-25T16:44:09.581 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:09.582 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 6.8 M 2024-03-25T16:44:09.582 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 33 M 2024-03-25T16:44:09.582 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:44:10.184 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 707 kB/s | 248 kB 00:00 2024-03-25T16:44:10.267 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-03-25T16:44:10.718 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 128 kB/s | 58 kB 00:00 2024-03-25T16:44:10.901 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 278 kB/s | 51 kB 00:00 2024-03-25T16:44:11.085 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 110 kB/s | 138 kB 00:01 2024-03-25T16:44:11.119 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 588 kB/s | 548 kB 00:00 2024-03-25T16:44:11.121 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:11.121 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.4 MB/s | 6.8 MB 00:01 2024-03-25T16:44:11.134 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:44:11.145 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:44:11.145 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:44:11.236 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:44:11.237 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:44:11.553 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:44:11.735 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:44:11.797 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:44:12.351 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:44:12.434 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:44:12.503 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:44:12.541 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:44:13.327 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:44:13.327 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:44:13.328 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:44:13.328 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:44:13.328 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:44:13.328 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:44:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:13.645 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:13.891 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-fuse 2024-03-25T16:44:14.470 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:15 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:14.626 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 964 k 2024-03-25T16:44:14.627 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 2.8 M 2024-03-25T16:44:14.628 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:44:14.812 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 426 kB/s | 79 kB 00:00 2024-03-25T16:44:17.639 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 294 kB/s | 885 kB 00:03 2024-03-25T16:44:17.643 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:17.643 INFO:teuthology.orchestra.run.smithi055.stdout:Total 320 kB/s | 964 kB 00:03 2024-03-25T16:44:17.643 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:44:17.665 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:44:17.665 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:44:17.742 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:44:17.743 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:44:17.932 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:44:18.084 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:44:18.156 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:44:18.835 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:44:18.835 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:44:19.126 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:44:19.127 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:19.127 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:19.127 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:44:19.127 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:19.127 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:19.445 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-volume 2024-03-25T16:44:20.023 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:21 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:20.178 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:20.179 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:20.179 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-03-25T16:44:20.179 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:20.179 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:20.179 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:44:20.179 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:44:20.179 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:44:20.179 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:20.180 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:20.180 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:20.180 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2024-03-25T16:44:20.180 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:20.180 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 332 k 2024-03-25T16:44:20.180 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 1.5 M 2024-03-25T16:44:20.180 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:44:20.358 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 388 kB/s | 69 kB 00:00 2024-03-25T16:44:21.533 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 195 kB/s | 264 kB 00:01 2024-03-25T16:44:21.534 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:21.535 INFO:teuthology.orchestra.run.smithi055.stdout:Total 245 kB/s | 332 kB 00:01 2024-03-25T16:44:21.535 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:44:21.546 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:44:21.546 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:44:21.588 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:44:21.588 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:44:21.747 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:44:21.942 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:44:21.986 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:44:22.001 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:44:22.001 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:44:22.001 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:44:22.001 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:22.744 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:44:23.085 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:44:23.085 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:23.085 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:23.085 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:44:23.086 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:44:23.086 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:23.086 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:23.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados-devel 2024-03-25T16:44:24.028 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:25 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:24.185 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 127 k 2024-03-25T16:44:24.186 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 456 k 2024-03-25T16:44:24.187 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:44:25.272 INFO:teuthology.orchestra.run.smithi055.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 117 kB/s | 127 kB 00:01 2024-03-25T16:44:25.273 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:25.273 INFO:teuthology.orchestra.run.smithi055.stdout:Total 116 kB/s | 127 kB 00:01 2024-03-25T16:44:25.273 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:44:25.281 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:44:25.281 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:44:25.344 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:44:25.344 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:44:25.500 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:44:25.600 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:26.023 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:26.351 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:26.351 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:26.351 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:26.351 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:26.351 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:26.351 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:26.578 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs2 2024-03-25T16:44:27.158 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:28 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:27.258 INFO:teuthology.orchestra.run.smithi055.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:44:27.313 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:27.314 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T16:44:27.314 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:27.381 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs-devel 2024-03-25T16:44:27.963 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:28 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:28.118 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:28.119 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:28.119 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T16:44:28.119 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:28.119 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:28.119 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:44:28.119 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:28.119 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:28.120 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:28.120 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T16:44:28.120 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:28.120 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 31 k 2024-03-25T16:44:28.120 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 140 k 2024-03-25T16:44:28.120 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:44:28.642 INFO:teuthology.orchestra.run.smithi055.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 60 kB/s | 31 kB 00:00 2024-03-25T16:44:28.643 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:28.643 INFO:teuthology.orchestra.run.smithi055.stdout:Total 59 kB/s | 31 kB 00:00 2024-03-25T16:44:28.643 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:44:28.647 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:44:28.647 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:44:28.674 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:44:28.674 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:44:28.796 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:44:28.902 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:29.264 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:29.534 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:29.534 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:29.534 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:29.535 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:29.535 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:29.535 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:29.694 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados2 2024-03-25T16:44:30.270 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:31 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:30.371 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:44:30.425 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:30.427 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T16:44:30.427 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:30.480 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librbd1 2024-03-25T16:44:31.077 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:32 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:31.180 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:44:31.235 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:31.236 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T16:44:31.236 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:31.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rados 2024-03-25T16:44:31.873 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:32 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:31.972 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:44:32.027 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:32.028 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T16:44:32.028 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:32.079 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rgw 2024-03-25T16:44:32.655 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:33 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:32.755 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:44:32.809 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:32.810 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T16:44:32.810 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:32.859 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-cephfs 2024-03-25T16:44:33.435 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:34 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:33.535 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:44:33.589 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:33.590 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T16:44:33.590 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:33.644 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rbd 2024-03-25T16:44:34.221 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:35 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:34.320 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:44:34.374 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:34.375 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T16:44:34.376 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:34.425 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-fuse 2024-03-25T16:44:35.001 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:36 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:35.156 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 85 k 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 231 k 2024-03-25T16:44:35.157 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:44:35.837 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 125 kB/s | 85 kB 00:00 2024-03-25T16:44:35.838 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:35.838 INFO:teuthology.orchestra.run.smithi055.stdout:Total 125 kB/s | 85 kB 00:00 2024-03-25T16:44:35.838 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:44:35.847 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:44:35.847 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:44:35.906 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:44:35.907 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:44:36.046 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:44:36.129 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:36.522 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:36.833 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:36.834 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:36.834 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:36.834 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:37.050 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-mirror 2024-03-25T16:44:37.623 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:38 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:37.778 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:37.778 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:37.778 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T16:44:37.778 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:37.778 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:37.778 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:44:37.778 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:37.778 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:37.778 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:37.779 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T16:44:37.779 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:37.779 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 3.1 M 2024-03-25T16:44:37.779 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 13 M 2024-03-25T16:44:37.779 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:44:38.865 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.8 MB/s | 3.1 MB 00:01 2024-03-25T16:44:38.866 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:38.866 INFO:teuthology.orchestra.run.smithi055.stdout:Total 2.8 MB/s | 3.1 MB 00:01 2024-03-25T16:44:38.866 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:44:38.878 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:44:38.878 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:44:38.957 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:44:38.958 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:44:39.409 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:44:39.457 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:39.482 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:39.482 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:44:39.482 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:44:39.482 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:44:39.482 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:44:39.482 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:40.598 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:40.598 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:40.598 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:40.599 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:40.599 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:40.599 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:40.836 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-nbd 2024-03-25T16:44:41.426 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:42 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:41.579 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T16:44:41.580 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:41.581 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 171 k 2024-03-25T16:44:41.581 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 494 k 2024-03-25T16:44:41.581 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:44:42.402 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 209 kB/s | 171 kB 00:00 2024-03-25T16:44:42.403 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:42.403 INFO:teuthology.orchestra.run.smithi055.stdout:Total 208 kB/s | 171 kB 00:00 2024-03-25T16:44:42.403 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:44:42.413 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:44:42.413 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:44:42.472 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:44:42.472 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:44:42.620 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:44:42.712 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:43.127 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:43.415 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:44:43.415 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:43.415 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:44:43.415 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:43.416 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:43.416 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:44:43.603 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:44:44.176 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:07:45 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:44:44.333 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:44:44.333 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:44.333 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T16:44:44.333 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 91 M 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 498 M 2024-03-25T16:44:44.334 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T16:45:00.364 INFO:teuthology.orchestra.run.smithi133.stdout:(13/13): python3-certifi-2021.10.8-2.el9.noarch 238 B/s | 15 kB 01:04 2024-03-25T16:45:00.366 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:00.367 INFO:teuthology.orchestra.run.smithi133.stdout:Total 32 kB/s | 2.0 MB 01:04 2024-03-25T16:45:00.367 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:45:00.381 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:45:00.381 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:45:00.476 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:45:00.477 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:45:00.987 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:45:01.151 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:45:01.214 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:45:01.270 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:45:01.307 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:45:01.348 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:45:01.450 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:45:01.568 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:45:01.616 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:45:02.348 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:45:02.405 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:45:02.439 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:45:02.486 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:45:02.508 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:45:03.225 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:45:03.225 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:45:03.225 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:45:03.225 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:45:03.225 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:45:03.225 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:45:03.226 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:45:03.226 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:45:03.226 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:45:03.226 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:45:03.226 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:45:03.226 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:45:03.226 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:45:03.583 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:45:03.584 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:45:03.585 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:03.585 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:03.826 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:45:04.386 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:16 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:04.536 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:04.537 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-03-25T16:45:04.537 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:45:04.537 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-03-25T16:45:04.537 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:45:04.537 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:45:04.537 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2024-03-25T16:45:04.538 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:04.539 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 6.8 M 2024-03-25T16:45:04.539 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 33 M 2024-03-25T16:45:04.539 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:45:05.465 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-03-25T16:45:06.081 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 225 kB/s | 248 kB 00:01 2024-03-25T16:45:06.140 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 812 kB/s | 548 kB 00:00 2024-03-25T16:45:06.257 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 437 kB/s | 51 kB 00:00 2024-03-25T16:45:06.415 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-03-25T16:45:06.824 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 75 kB/s | 138 kB 00:01 2024-03-25T16:45:06.825 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:06.825 INFO:teuthology.orchestra.run.smithi133.stdout:Total 3.0 MB/s | 6.8 MB 00:02 2024-03-25T16:45:06.840 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:45:06.851 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:45:06.851 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:45:06.942 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:45:06.943 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:45:07.241 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:45:07.410 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:45:07.458 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:45:07.963 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:45:08.052 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:45:08.122 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:45:08.154 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:45:08.934 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:45:08.934 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:45:08.934 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:45:08.934 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:45:08.934 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:45:08.934 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:45:09.221 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:45:09.221 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:09.222 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:45:09.222 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:45:09.222 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:45:09.222 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:45:09.222 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:45:09.222 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:45:09.222 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:45:09.222 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:09.222 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:09.470 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-fuse 2024-03-25T16:45:10.038 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:22 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:10.189 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:10.190 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:10.190 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:45:10.190 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:10.190 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:45:10.190 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:45:10.190 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:45:10.190 INFO:teuthology.orchestra.run.smithi133.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:45:10.191 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:10.191 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:45:10.191 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:10.191 INFO:teuthology.orchestra.run.smithi133.stdout:Install 2 Packages 2024-03-25T16:45:10.191 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:10.191 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 964 k 2024-03-25T16:45:10.191 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 2.8 M 2024-03-25T16:45:10.191 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:45:10.392 INFO:teuthology.orchestra.run.smithi133.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 393 kB/s | 79 kB 00:00 2024-03-25T16:45:15.158 INFO:teuthology.orchestra.run.smithi133.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 178 kB/s | 885 kB 00:04 2024-03-25T16:45:15.160 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:15.160 INFO:teuthology.orchestra.run.smithi133.stdout:Total 194 kB/s | 964 kB 00:04 2024-03-25T16:45:15.160 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:45:15.184 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:45:15.184 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:45:15.263 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:45:15.263 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:45:15.436 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:45:15.576 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:45:15.641 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:45:16.329 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:45:16.329 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:45:16.640 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:45:16.640 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:16.640 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:45:16.640 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:45:16.640 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:16.640 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:16.923 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-volume 2024-03-25T16:45:17.500 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:29 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:17.655 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:17.655 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:17.655 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-25T16:45:17.655 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:17.655 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:45:17.655 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:45:17.655 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:45:17.656 INFO:teuthology.orchestra.run.smithi133.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:45:17.656 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:17.656 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:45:17.656 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:17.656 INFO:teuthology.orchestra.run.smithi133.stdout:Install 2 Packages 2024-03-25T16:45:17.656 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:17.656 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 332 k 2024-03-25T16:45:17.656 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.5 M 2024-03-25T16:45:17.657 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:45:17.837 INFO:teuthology.orchestra.run.smithi133.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 382 kB/s | 69 kB 00:00 2024-03-25T16:45:19.737 INFO:teuthology.orchestra.run.smithi133.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 127 kB/s | 264 kB 00:02 2024-03-25T16:45:19.739 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:19.739 INFO:teuthology.orchestra.run.smithi133.stdout:Total 160 kB/s | 332 kB 00:02 2024-03-25T16:45:19.739 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:45:19.750 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:45:19.750 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:45:19.797 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:45:19.798 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:45:19.953 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:45:20.137 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:45:20.181 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:45:20.197 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:45:20.197 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:45:20.197 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:45:20.197 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:20.944 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:45:21.298 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:45:21.298 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:21.298 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:45:21.298 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:45:21.298 INFO:teuthology.orchestra.run.smithi133.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:45:21.298 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:21.298 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:21.619 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados-devel 2024-03-25T16:45:22.193 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:34 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:22.343 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:22.343 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:22.343 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T16:45:22.343 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 127 k 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 456 k 2024-03-25T16:45:22.344 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:45:23.510 INFO:teuthology.orchestra.run.smithi133.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 109 kB/s | 127 kB 00:01 2024-03-25T16:45:23.510 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:23.510 INFO:teuthology.orchestra.run.smithi133.stdout:Total 108 kB/s | 127 kB 00:01 2024-03-25T16:45:23.511 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:45:23.517 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:45:23.517 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:45:23.576 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:45:23.577 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:45:23.728 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:45:23.842 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:24.336 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:24.630 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:24.630 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:24.630 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:45:24.630 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:24.630 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:24.630 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:24.848 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs2 2024-03-25T16:45:25.421 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:37 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:25.520 INFO:teuthology.orchestra.run.smithi133.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:45:25.573 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:25.574 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T16:45:25.575 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:25.639 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs-devel 2024-03-25T16:45:26.207 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:38 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:26.355 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:26.356 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:26.356 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T16:45:26.356 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:26.356 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:45:26.356 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:45:26.356 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:26.356 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:45:26.356 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:26.356 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T16:45:26.357 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:26.357 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 31 k 2024-03-25T16:45:26.357 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 140 k 2024-03-25T16:45:26.357 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:45:26.979 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 50 kB/s | 31 kB 00:00 2024-03-25T16:45:26.979 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:26.980 INFO:teuthology.orchestra.run.smithi133.stdout:Total 50 kB/s | 31 kB 00:00 2024-03-25T16:45:26.980 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:45:26.983 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:45:26.983 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:45:27.018 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:45:27.019 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:45:27.140 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:45:27.245 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:27.646 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:27.920 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:27.920 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:27.920 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:45:27.920 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:27.920 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:27.920 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:28.081 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados2 2024-03-25T16:45:28.652 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:40 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:28.750 INFO:teuthology.orchestra.run.smithi133.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:45:28.801 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:28.802 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T16:45:28.803 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:28.848 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librbd1 2024-03-25T16:45:29.419 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:41 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:29.516 INFO:teuthology.orchestra.run.smithi133.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:45:29.567 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:29.568 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T16:45:29.568 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:29.611 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rados 2024-03-25T16:45:30.178 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:42 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:30.275 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:45:30.326 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:30.327 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T16:45:30.327 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:30.377 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rgw 2024-03-25T16:45:30.946 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:42 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:31.042 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:45:31.094 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:31.095 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T16:45:31.096 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:31.145 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-cephfs 2024-03-25T16:45:31.713 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:43 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:31.810 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:45:31.861 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:31.862 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T16:45:31.862 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:31.910 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rbd 2024-03-25T16:45:32.478 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:44 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:32.576 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:45:32.630 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:32.631 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T16:45:32.631 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:32.680 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-fuse 2024-03-25T16:45:33.287 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:45 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:33.443 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:33.444 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:33.444 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2024-03-25T16:45:33.444 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:33.444 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:45:33.444 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:45:33.444 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:33.445 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:45:33.445 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:33.445 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T16:45:33.445 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:33.445 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 85 k 2024-03-25T16:45:33.445 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 231 k 2024-03-25T16:45:33.445 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:45:34.168 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 118 kB/s | 85 kB 00:00 2024-03-25T16:45:34.169 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:34.169 INFO:teuthology.orchestra.run.smithi133.stdout:Total 118 kB/s | 85 kB 00:00 2024-03-25T16:45:34.169 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:45:34.176 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:45:34.176 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:45:34.233 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:45:34.233 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:45:34.371 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:45:34.483 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:34.920 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:35.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:35.212 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:35.212 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:45:35.212 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:35.212 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:35.212 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:35.436 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-mirror 2024-03-25T16:45:36.008 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:08:48 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:36.160 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:36.160 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:36.160 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T16:45:36.160 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 3.1 M 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 13 M 2024-03-25T16:45:36.161 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:45:48.635 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 253 kB/s | 3.1 MB 00:12 2024-03-25T16:45:48.636 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:48.636 INFO:teuthology.orchestra.run.smithi133.stdout:Total 253 kB/s | 3.1 MB 00:12 2024-03-25T16:45:48.637 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:45:48.648 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:45:48.648 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:45:48.724 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:45:48.724 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:45:49.166 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:45:49.227 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:49.252 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:49.252 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:45:49.252 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:45:49.252 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:45:49.252 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:45:49.252 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:50.364 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:50.364 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:50.364 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:45:50.364 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:50.364 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:50.364 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:50.603 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-nbd 2024-03-25T16:45:51.183 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:09:03 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:51.333 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:51.334 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:51.335 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2024-03-25T16:45:51.335 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:51.335 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:45:51.335 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:45:51.335 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:51.335 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:45:51.336 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:51.336 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T16:45:51.336 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:51.336 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 171 k 2024-03-25T16:45:51.336 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 494 k 2024-03-25T16:45:51.336 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:45:53.021 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 102 kB/s | 171 kB 00:01 2024-03-25T16:45:53.022 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:53.022 INFO:teuthology.orchestra.run.smithi133.stdout:Total 102 kB/s | 171 kB 00:01 2024-03-25T16:45:53.022 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:45:53.032 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:45:53.032 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:45:53.092 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:45:53.093 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:45:53.240 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:45:53.352 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:53.784 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:54.066 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:45:54.066 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:54.066 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:45:54.066 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:54.066 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:54.066 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:45:54.263 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:45:54.834 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:09:06 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:45:54.984 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:45:54.984 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:54.984 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T16:45:54.984 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:54.984 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:45:54.985 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:45:54.985 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-25T16:45:54.985 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:45:54.985 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:54.985 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:45:54.985 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:45:54.985 INFO:teuthology.orchestra.run.smithi133.stdout:Install 2 Packages 2024-03-25T16:45:54.985 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:45:54.986 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 91 M 2024-03-25T16:45:54.986 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 498 M 2024-03-25T16:45:54.986 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:46:26.362 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 225 kB/s | 22 MB 01:42 2024-03-25T16:47:05.173 INFO:teuthology.orchestra.run.smithi133.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 327 kB/s | 22 MB 01:10 2024-03-25T16:48:50.673 INFO:teuthology.orchestra.run.smithi133.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 397 kB/s | 68 MB 02:55 2024-03-25T16:48:50.674 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:48:50.674 INFO:teuthology.orchestra.run.smithi133.stdout:Total 528 kB/s | 91 MB 02:55 2024-03-25T16:48:50.674 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T16:48:50.679 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T16:48:50.679 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T16:48:51.261 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T16:48:51.262 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T16:48:54.347 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T16:49:01.479 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:49:01.632 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:49:04.932 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:49:04.932 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:49:05.198 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:49:05.198 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:49:05.198 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T16:49:05.198 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:49:05.198 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:49:05.198 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:49:05.198 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T16:49:05.449 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:49:06.027 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:12:18 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T16:49:06.188 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T16:49:06.189 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T16:49:06.190 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 416 M 2024-03-25T16:49:06.190 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.7 G 2024-03-25T16:49:06.190 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T16:49:49.708 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 228 kB/s | 68 MB 05:05 2024-03-25T16:49:49.709 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:49:49.710 INFO:teuthology.orchestra.run.smithi055.stdout:Total 304 kB/s | 91 MB 05:05 2024-03-25T16:49:49.710 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T16:49:49.714 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T16:49:49.714 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T16:49:50.326 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T16:49:50.326 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T16:49:53.598 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T16:50:01.109 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:50:01.186 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:50:04.576 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:50:04.576 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:50:04.848 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:50:04.849 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:50:04.849 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T16:50:04.849 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:04.849 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:04.849 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:50:04.849 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T16:50:05.073 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:50:05.647 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:13:06 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T16:50:05.805 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T16:50:05.806 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T16:50:05.807 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 416 M 2024-03-25T16:50:05.807 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 1.7 G 2024-03-25T16:50:05.807 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T17:13:54.315 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 286 kB/s | 416 MB 24:48 2024-03-25T17:13:54.315 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:54.315 INFO:teuthology.orchestra.run.smithi133.stdout:Total 286 kB/s | 416 MB 24:48 2024-03-25T17:13:54.316 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T17:13:54.319 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T17:13:54.319 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T17:13:56.791 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T17:13:56.822 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T17:14:39.171 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T17:14:39.257 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:14:47.062 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:14:47.320 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:14:47.321 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:14:47.321 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T17:14:47.321 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:14:47.321 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:14:47.321 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T17:14:47.555 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:14:48.123 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:38:00 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T17:14:48.270 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T17:14:48.271 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-03-25T17:14:48.271 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T17:14:48.271 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-03-25T17:14:48.271 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T17:14:48.271 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:14:48.272 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:14:48.272 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T17:14:48.272 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-03-25T17:14:48.272 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T17:14:48.272 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:14:48.272 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 1.4 M 2024-03-25T17:14:48.272 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 6.4 M 2024-03-25T17:14:48.272 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T17:14:53.593 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 275 kB/s | 1.4 MB 00:05 2024-03-25T17:14:53.594 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:53.595 INFO:teuthology.orchestra.run.smithi133.stdout:Total 274 kB/s | 1.4 MB 00:05 2024-03-25T17:14:53.595 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T17:14:53.599 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T17:14:53.599 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T17:14:53.644 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T17:14:53.644 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T17:14:53.903 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T17:14:54.008 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:14:54.441 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:14:54.722 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:14:54.723 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:14:54.723 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T17:14:54.723 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:14:54.723 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:14:54.723 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T17:14:54.922 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:14:55.485 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:38:07 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T17:14:55.635 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T17:14:55.636 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:14:55.636 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T17:14:55.636 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 176 M 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 753 M 2024-03-25T17:14:55.637 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T17:15:04.671 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 284 kB/s | 416 MB 24:58 2024-03-25T17:15:04.671 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:04.672 INFO:teuthology.orchestra.run.smithi055.stdout:Total 284 kB/s | 416 MB 24:58 2024-03-25T17:15:04.672 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T17:15:04.676 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T17:15:04.676 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T17:15:07.265 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T17:15:07.265 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T17:15:51.916 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T17:15:52.019 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:16:00.374 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:16:00.654 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:16:00.655 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:16:00.655 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T17:16:00.655 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:00.655 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:16:00.655 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T17:16:00.889 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:16:01.467 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:39:02 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T17:16:01.623 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T17:16:01.624 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-25T17:16:01.624 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T17:16:01.624 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-25T17:16:01.624 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T17:16:01.624 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:16:01.624 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:16:01.624 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T17:16:01.624 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2024-03-25T17:16:01.624 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T17:16:01.625 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:16:01.625 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 1.4 M 2024-03-25T17:16:01.625 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 6.4 M 2024-03-25T17:16:01.625 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T17:16:07.078 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 268 kB/s | 1.4 MB 00:05 2024-03-25T17:16:07.079 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:16:07.079 INFO:teuthology.orchestra.run.smithi055.stdout:Total 268 kB/s | 1.4 MB 00:05 2024-03-25T17:16:07.079 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T17:16:07.084 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T17:16:07.084 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T17:16:07.130 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T17:16:07.131 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T17:16:07.398 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T17:16:07.504 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:16:07.910 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:16:08.202 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:16:08.202 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:16:08.203 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T17:16:08.203 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:08.203 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:16:08.203 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T17:16:08.384 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:16:08.963 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:39:09 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T17:16:09.117 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T17:16:09.118 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T17:16:09.118 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T17:16:09.118 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T17:16:09.118 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T17:16:09.118 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:16:09.119 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:16:09.119 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T17:16:09.119 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T17:16:09.119 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T17:16:09.119 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:16:09.119 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 176 M 2024-03-25T17:16:09.119 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 753 M 2024-03-25T17:16:09.119 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T17:22:46.791 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 382 kB/s | 176 MB 07:51 2024-03-25T17:22:46.817 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:22:46.818 INFO:teuthology.orchestra.run.smithi133.stdout:Total 382 kB/s | 176 MB 07:51 2024-03-25T17:22:46.818 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T17:22:46.818 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T17:22:46.818 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T17:22:47.855 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T17:22:47.855 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T17:23:06.064 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T17:23:06.237 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:23:10.818 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:23:11.074 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:23:11.074 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:23:11.074 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T17:23:11.074 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:23:11.074 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:23:11.074 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T17:23:11.293 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:23:11.857 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:46:23 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T17:23:12.007 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T17:23:12.007 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:23:12.007 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 672 M 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 3.1 G 2024-03-25T17:23:12.008 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T17:25:31.947 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 320 kB/s | 176 MB 09:22 2024-03-25T17:25:31.949 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:31.949 INFO:teuthology.orchestra.run.smithi055.stdout:Total 320 kB/s | 176 MB 09:22 2024-03-25T17:25:31.949 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T17:25:31.951 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T17:25:31.951 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T17:25:33.060 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T17:25:33.061 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T17:25:52.099 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T17:25:52.396 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:25:57.157 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:25:57.428 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:25:57.428 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:25:57.428 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T17:25:57.428 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:57.428 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:25:57.428 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T17:25:57.751 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:25:58.324 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:48:59 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T17:25:58.481 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T17:25:58.482 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T17:25:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T17:25:58.482 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T17:25:58.482 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T17:25:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:25:58.482 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:25:58.482 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T17:25:58.482 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T17:25:58.483 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T17:25:58.483 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:25:58.483 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 672 M 2024-03-25T17:25:58.483 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 3.1 G 2024-03-25T17:25:58.483 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T17:54:13.811 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 370 kB/s | 672 MB 31:01 2024-03-25T17:54:13.826 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:13.826 INFO:teuthology.orchestra.run.smithi133.stdout:Total 370 kB/s | 672 MB 31:01 2024-03-25T17:54:13.826 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T17:54:13.827 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T17:54:13.827 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T17:54:17.773 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T17:54:17.773 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T17:55:28.615 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T17:55:28.718 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:36.304 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:36.559 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:36.560 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:55:36.560 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T17:55:36.561 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:36.561 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:55:36.561 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T17:55:36.845 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:55:37.415 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:18:49 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T17:55:37.511 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:55:37.564 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T17:55:37.565 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-03-25T17:55:37.565 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T17:55:37.637 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:55:38.228 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:18:50 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T17:55:38.376 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T17:55:38.377 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:55:38.377 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T17:55:38.377 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:55:38.377 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T17:55:38.378 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:55:38.378 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:55:38.378 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T17:55:38.378 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:55:38.378 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T17:55:38.378 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:55:38.378 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 20 M 2024-03-25T17:55:38.378 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 103 M 2024-03-25T17:55:38.378 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T17:56:26.105 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 423 kB/s | 20 MB 00:47 2024-03-25T17:56:26.106 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:26.107 INFO:teuthology.orchestra.run.smithi133.stdout:Total 423 kB/s | 20 MB 00:47 2024-03-25T17:56:26.107 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T17:56:26.110 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T17:56:26.110 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T17:56:26.260 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T17:56:26.260 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T17:56:28.497 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T17:56:28.803 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:29.761 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:30.015 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:30.015 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:56:30.015 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T17:56:30.016 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:30.016 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:56:30.016 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T17:56:30.199 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:56:30.765 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:19:42 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T17:56:30.913 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T17:56:30.913 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 48 M 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 173 M 2024-03-25T17:56:30.914 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T17:57:41.828 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 362 kB/s | 672 MB 31:43 2024-03-25T17:57:41.829 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:41.829 INFO:teuthology.orchestra.run.smithi055.stdout:Total 362 kB/s | 672 MB 31:43 2024-03-25T17:57:41.829 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T17:57:41.833 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T17:57:41.833 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T17:57:45.985 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T17:57:45.985 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T17:58:46.862 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 359 kB/s | 48 MB 02:15 2024-03-25T17:58:46.862 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:46.863 INFO:teuthology.orchestra.run.smithi133.stdout:Total 359 kB/s | 48 MB 02:15 2024-03-25T17:58:46.863 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T17:58:46.865 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T17:58:46.865 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T17:58:47.178 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T17:58:47.178 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T17:58:51.922 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T17:58:52.101 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:53.435 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:53.691 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:53.691 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:58:53.692 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T17:58:53.692 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:53.692 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:58:53.692 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T17:58:53.883 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:58:54.454 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:22:06 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T17:58:54.607 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T17:58:54.608 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:58:54.608 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T17:58:54.608 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:58:54.608 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T17:58:54.609 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:58:54.609 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:58:54.609 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T17:58:54.609 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T17:58:54.609 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T17:58:54.609 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T17:58:54.609 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 99 M 2024-03-25T17:58:54.609 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 449 M 2024-03-25T17:58:54.609 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T17:59:00.841 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T17:59:00.991 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:59:08.923 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:59:09.195 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:59:09.195 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:59:09.195 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T17:59:09.195 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:09.195 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:59:09.195 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T17:59:09.475 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:59:10.052 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:22:11 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T17:59:10.152 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:59:10.206 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T17:59:10.208 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2024-03-25T17:59:10.208 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T17:59:10.276 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:59:10.853 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:22:11 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T17:59:11.007 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T17:59:11.008 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T17:59:11.008 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T17:59:11.008 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T17:59:11.008 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T17:59:11.008 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:59:11.009 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:59:11.009 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T17:59:11.009 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T17:59:11.009 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T17:59:11.009 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T17:59:11.009 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 20 M 2024-03-25T17:59:11.009 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 103 M 2024-03-25T17:59:11.009 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:00:19.328 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 295 kB/s | 20 MB 01:08 2024-03-25T18:00:19.328 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:19.329 INFO:teuthology.orchestra.run.smithi055.stdout:Total 295 kB/s | 20 MB 01:08 2024-03-25T18:00:19.329 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:00:19.332 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:00:19.332 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:00:19.488 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:00:19.488 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:00:21.847 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:00:21.937 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:22.880 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:23.145 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:23.145 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:00:23.145 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:00:23.145 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.145 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:00:23.145 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:00:23.328 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:00:23.906 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:23:24 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:00:24.060 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:00:24.061 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:00:24.061 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:00:24.061 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:00:24.061 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:00:24.061 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:00:24.062 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:00:24.062 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:00:24.062 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:00:24.062 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:00:24.062 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:00:24.062 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 48 M 2024-03-25T18:00:24.062 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 173 M 2024-03-25T18:00:24.062 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:02:43.667 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 350 kB/s | 48 MB 02:19 2024-03-25T18:02:43.668 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:43.668 INFO:teuthology.orchestra.run.smithi055.stdout:Total 350 kB/s | 48 MB 02:19 2024-03-25T18:02:43.668 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:02:43.671 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:02:43.671 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:02:43.998 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:02:43.998 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:02:49.045 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:02:49.142 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:02:50.515 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:02:50.785 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:02:50.785 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:02:50.786 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:02:50.786 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:50.786 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:02:50.786 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:02:50.991 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:02:51.567 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:25:52 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:02:51.721 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:02:51.722 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:02:51.722 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:02:51.722 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 99 M 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 449 M 2024-03-25T18:02:51.723 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:05:14.365 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 266 kB/s | 99 MB 06:19 2024-03-25T18:05:14.365 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:14.365 INFO:teuthology.orchestra.run.smithi133.stdout:Total 266 kB/s | 99 MB 06:19 2024-03-25T18:05:14.365 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:05:14.368 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:05:14.368 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:05:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:05:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:05:25.228 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:05:25.335 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:28.183 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:28.445 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:28.445 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:05:28.445 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:05:28.446 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:28.446 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:05:28.446 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:05:28.624 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:05:29.192 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:28:41 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:05:29.339 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:05:29.340 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:05:29.340 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:05:29.340 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:05:29.340 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:05:29.340 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:05:29.340 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:05:29.340 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:05:29.341 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:05:29.341 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:05:29.341 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:05:29.341 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 287 M 2024-03-25T18:05:29.341 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.2 G 2024-03-25T18:05:29.341 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:08:03.012 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 324 kB/s | 99 MB 05:11 2024-03-25T18:08:03.012 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:03.012 INFO:teuthology.orchestra.run.smithi055.stdout:Total 324 kB/s | 99 MB 05:11 2024-03-25T18:08:03.012 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:08:03.015 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:08:03.015 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:08:03.652 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:08:03.653 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:08:14.546 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:08:14.653 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:08:17.775 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:08:18.041 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:08:18.042 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:08:18.042 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:08:18.042 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:18.042 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:08:18.042 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:08:18.225 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:08:18.801 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:31:19 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:08:18.956 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:08:18.956 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:08:18.956 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:08:18.956 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:08:18.957 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:08:18.957 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:08:18.957 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:08:18.957 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:08:18.957 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:08:18.957 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:08:18.957 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:08:18.957 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 287 M 2024-03-25T18:08:18.957 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 1.2 G 2024-03-25T18:08:18.958 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:18:06.966 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 388 kB/s | 287 MB 12:37 2024-03-25T18:18:06.966 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:06.967 INFO:teuthology.orchestra.run.smithi133.stdout:Total 388 kB/s | 287 MB 12:37 2024-03-25T18:18:06.967 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:18:06.969 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:18:06.969 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:18:08.675 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:18:08.675 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:18:38.025 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:18:38.117 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:45.400 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:45.656 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:45.657 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:18:45.657 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:18:45.657 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:45.657 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:18:45.657 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:18:45.872 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:18:46.438 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:41:58 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:18:46.588 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:18:46.588 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:18:46.588 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:18:46.588 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:18:46.589 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:18:46.589 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:18:46.589 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:18:46.589 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:18:46.589 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:18:46.589 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:18:46.589 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:18:46.589 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 5.9 M 2024-03-25T18:18:46.590 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 28 M 2024-03-25T18:18:46.590 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:19:04.352 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 343 kB/s | 5.9 MB 00:17 2024-03-25T18:19:04.352 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:04.353 INFO:teuthology.orchestra.run.smithi133.stdout:Total 343 kB/s | 5.9 MB 00:17 2024-03-25T18:19:04.353 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:19:04.357 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:19:04.357 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:19:04.428 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:19:04.429 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:19:05.155 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:19:05.259 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:05.780 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:06.037 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:06.037 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:06.038 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:19:06.038 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:06.038 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:06.038 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:19:06.228 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:19:06.791 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:42:18 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:19:06.942 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:19:06.942 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:19:06.943 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:19:06.943 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:19:06.943 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:19:06.943 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:19:06.943 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:06.943 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:19:06.943 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:19:06.943 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:19:06.943 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:06.944 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 109 k 2024-03-25T18:19:06.944 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 440 k 2024-03-25T18:19:06.944 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:19:07.721 INFO:teuthology.orchestra.run.smithi133.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 140 kB/s | 109 kB 00:00 2024-03-25T18:19:07.721 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:07.722 INFO:teuthology.orchestra.run.smithi133.stdout:Total 140 kB/s | 109 kB 00:00 2024-03-25T18:19:07.722 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:19:07.726 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:19:07.726 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:19:07.765 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:19:07.765 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:19:07.887 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:19:07.986 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:19:08.389 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:19:08.652 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:19:08.653 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:08.653 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:19:08.653 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:08.653 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:08.653 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:19:08.820 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:19:09.383 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:42:21 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:19:09.532 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:19:09.533 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:19:09.533 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:19:09.533 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:19:09.533 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:19:09.534 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:19:09.534 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:09.534 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:19:09.534 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:19:09.534 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:19:09.534 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:09.534 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 5.4 M 2024-03-25T18:19:09.534 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 25 M 2024-03-25T18:19:09.534 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:19:23.080 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 405 kB/s | 5.4 MB 00:13 2024-03-25T18:19:23.081 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:23.081 INFO:teuthology.orchestra.run.smithi133.stdout:Total 405 kB/s | 5.4 MB 00:13 2024-03-25T18:19:23.082 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:19:23.086 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:19:23.086 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:19:23.157 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:19:23.157 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:19:23.829 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:19:23.936 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:19:24.431 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:19:24.701 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:19:24.701 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:24.701 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:19:24.701 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:24.702 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:24.702 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:19:24.880 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados2-debuginfo 2024-03-25T18:19:25.459 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:42:37 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:19:25.618 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 44 M 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 253 M 2024-03-25T18:19:25.619 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:20:50.363 INFO:teuthology.orchestra.run.smithi133.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 535 kB/s | 44 MB 01:24 2024-03-25T18:20:50.364 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:50.364 INFO:teuthology.orchestra.run.smithi133.stdout:Total 535 kB/s | 44 MB 01:24 2024-03-25T18:20:50.364 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:20:50.367 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:20:50.367 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:20:50.667 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:20:50.668 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:20:55.726 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:20:55.828 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:20:57.664 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:20:57.935 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:20:57.935 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:20:57.935 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:20:57.935 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:57.935 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:20:57.935 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:20:58.119 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:20:58.686 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:44:10 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:20:58.837 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:20:58.837 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 52 M 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 308 M 2024-03-25T18:20:58.838 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:21:12.889 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 379 kB/s | 287 MB 12:53 2024-03-25T18:21:12.889 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:12.890 INFO:teuthology.orchestra.run.smithi055.stdout:Total 379 kB/s | 287 MB 12:53 2024-03-25T18:21:12.890 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:21:12.894 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:21:12.894 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:21:14.683 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:21:14.683 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:21:45.770 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:21:45.880 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:53.634 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:53.927 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:53.927 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:21:53.927 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:21:53.927 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:53.927 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:21:53.927 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:21:54.151 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:21:54.729 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:44:55 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:21:54.883 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:21:54.884 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:21:54.884 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:21:54.884 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:21:54.884 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:21:54.884 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:21:54.885 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:21:54.885 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:21:54.885 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:21:54.885 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:21:54.885 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:21:54.885 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 5.9 M 2024-03-25T18:21:54.885 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 28 M 2024-03-25T18:21:54.885 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:22:13.279 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 331 kB/s | 5.9 MB 00:18 2024-03-25T18:22:13.280 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:13.280 INFO:teuthology.orchestra.run.smithi055.stdout:Total 331 kB/s | 5.9 MB 00:18 2024-03-25T18:22:13.280 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:22:13.285 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:22:13.285 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:22:13.358 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:22:13.359 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:22:14.130 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:22:14.239 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:22:14.771 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:22:15.053 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:22:15.053 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:15.053 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:22:15.053 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:15.053 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:15.053 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:22:15.238 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:22:15.809 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:45:16 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:22:15.964 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:22:15.964 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:22:15.964 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:22:15.964 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:22:15.964 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:22:15.965 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:22:15.965 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:15.965 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:22:15.965 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:22:15.965 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:22:15.965 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:15.965 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 109 k 2024-03-25T18:22:15.965 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 440 k 2024-03-25T18:22:15.965 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:22:16.881 INFO:teuthology.orchestra.run.smithi055.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 119 kB/s | 109 kB 00:00 2024-03-25T18:22:16.882 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:16.882 INFO:teuthology.orchestra.run.smithi055.stdout:Total 118 kB/s | 109 kB 00:00 2024-03-25T18:22:16.883 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:22:16.887 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:22:16.887 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:22:16.927 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:22:16.927 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:22:17.051 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:22:17.276 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:22:17.651 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:22:17.928 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:22:17.928 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:17.928 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:22:17.928 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:17.928 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:17.928 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:22:18.241 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:22:18.816 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:45:19 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:22:18.970 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:22:18.970 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 5.4 M 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 25 M 2024-03-25T18:22:18.971 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:22:26.384 INFO:teuthology.orchestra.run.smithi055.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 739 kB/s | 5.4 MB 00:07 2024-03-25T18:22:26.385 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:26.386 INFO:teuthology.orchestra.run.smithi055.stdout:Total 739 kB/s | 5.4 MB 00:07 2024-03-25T18:22:26.386 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:22:26.390 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:22:26.390 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:22:26.461 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:22:26.462 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:22:27.159 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:22:27.271 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:22:27.798 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:22:28.069 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:22:28.069 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:28.069 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:22:28.069 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:28.069 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:28.069 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:22:28.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados2-debuginfo 2024-03-25T18:22:28.832 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:45:29 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:22:28.987 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:22:28.987 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:22:28.988 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:22:28.988 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:22:28.988 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:22:28.988 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:22:28.988 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:28.988 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:22:28.988 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:22:28.988 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:22:28.988 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:22:28.989 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 44 M 2024-03-25T18:22:28.989 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 253 M 2024-03-25T18:22:28.989 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:22:54.581 INFO:teuthology.orchestra.run.smithi133.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 459 kB/s | 52 MB 01:55 2024-03-25T18:22:54.582 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:54.582 INFO:teuthology.orchestra.run.smithi133.stdout:Total 459 kB/s | 52 MB 01:55 2024-03-25T18:22:54.582 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:22:54.586 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:22:54.586 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:22:54.930 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:22:54.930 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:23:00.919 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:23:01.164 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:03.213 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:03.470 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:03.470 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:03.470 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:23:03.470 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:03.470 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:03.470 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:23:03.667 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:23:04.230 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:46:16 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:23:04.380 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:23:04.381 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:04.382 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 615 k 2024-03-25T18:23:04.382 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 2.1 M 2024-03-25T18:23:04.382 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:23:06.216 INFO:teuthology.orchestra.run.smithi133.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 335 kB/s | 615 kB 00:01 2024-03-25T18:23:06.217 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:06.217 INFO:teuthology.orchestra.run.smithi133.stdout:Total 335 kB/s | 615 kB 00:01 2024-03-25T18:23:06.217 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:23:06.221 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:23:06.222 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:23:06.264 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:23:06.264 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:23:06.429 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:23:06.549 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:23:06.987 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:23:07.235 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:23:07.235 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:07.236 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:23:07.236 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:07.236 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:07.236 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:23:07.464 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:23:08.028 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:46:20 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:23:08.179 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:23:08.180 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:08.180 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:23:08.180 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:08.180 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:23:08.180 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:23:08.180 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:08.180 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:23:08.180 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:08.180 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:23:08.181 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:08.181 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 771 k 2024-03-25T18:23:08.181 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 2.7 M 2024-03-25T18:23:08.181 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:23:11.344 INFO:teuthology.orchestra.run.smithi133.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 244 kB/s | 771 kB 00:03 2024-03-25T18:23:11.344 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:11.345 INFO:teuthology.orchestra.run.smithi133.stdout:Total 244 kB/s | 771 kB 00:03 2024-03-25T18:23:11.345 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:23:11.349 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:23:11.349 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:23:11.388 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:23:11.388 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:23:11.567 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:23:11.659 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:23:12.043 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:23:12.294 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:23:12.294 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:12.294 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:23:12.294 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:12.294 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:12.295 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:23:12.499 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:23:13.065 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:46:25 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:23:13.214 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:23:13.215 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:13.216 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 1.1 M 2024-03-25T18:23:13.216 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 4.3 M 2024-03-25T18:23:13.216 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:23:16.588 INFO:teuthology.orchestra.run.smithi133.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 341 kB/s | 1.1 MB 00:03 2024-03-25T18:23:16.589 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:16.589 INFO:teuthology.orchestra.run.smithi133.stdout:Total 341 kB/s | 1.1 MB 00:03 2024-03-25T18:23:16.590 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:23:16.593 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:23:16.594 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:23:16.636 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:23:16.637 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:23:16.882 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:23:16.973 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:17.373 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:17.634 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:17.634 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:17.634 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:23:17.634 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:17.634 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:17.635 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:23:17.831 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:23:18.395 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:46:30 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:23:18.546 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:23:18.547 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:18.547 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:23:18.547 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:18.547 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:23:18.547 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:23:18.547 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:18.547 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:23:18.547 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:18.547 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:23:18.548 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:18.548 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 337 k 2024-03-25T18:23:18.548 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 1.0 M 2024-03-25T18:23:18.548 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:23:19.824 INFO:teuthology.orchestra.run.smithi133.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 264 kB/s | 337 kB 00:01 2024-03-25T18:23:19.825 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:19.825 INFO:teuthology.orchestra.run.smithi133.stdout:Total 264 kB/s | 337 kB 00:01 2024-03-25T18:23:19.826 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:23:19.830 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:23:19.830 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:23:19.870 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:23:19.870 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:23:20.008 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:23:20.123 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:20.478 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:20.734 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:20.734 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:20.735 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:23:20.735 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:20.735 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:20.735 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:23:20.938 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:23:21.500 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:46:33 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:23:21.649 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:23:21.650 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:21.651 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 509 k 2024-03-25T18:23:21.651 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 2.5 M 2024-03-25T18:23:21.651 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:23:23.102 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 351 kB/s | 509 kB 00:01 2024-03-25T18:23:23.103 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:23.103 INFO:teuthology.orchestra.run.smithi133.stdout:Total 350 kB/s | 509 kB 00:01 2024-03-25T18:23:23.104 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:23:23.107 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:23:23.108 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:23:23.146 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:23:23.147 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:23:23.305 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:23:23.401 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:23:23.785 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:23:24.042 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:23:24.042 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:24.042 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:23:24.042 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:24.042 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:24.042 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:23:24.240 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:23:24.802 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:46:36 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:23:24.952 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:23:24.952 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:24.952 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:23:24.953 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:24.953 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:23:24.953 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:23:24.953 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:24.953 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:23:24.953 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:23:24.953 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:23:24.953 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:23:24.954 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 52 M 2024-03-25T18:23:24.954 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 298 M 2024-03-25T18:23:24.954 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:24:17.112 INFO:teuthology.orchestra.run.smithi055.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 419 kB/s | 44 MB 01:48 2024-03-25T18:24:17.113 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:17.113 INFO:teuthology.orchestra.run.smithi055.stdout:Total 419 kB/s | 44 MB 01:48 2024-03-25T18:24:17.113 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:24:17.117 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:24:17.117 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:24:17.430 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:24:17.431 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:24:22.771 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:24:22.864 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:24:24.737 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:24:25.012 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:24:25.012 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:24:25.012 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:24:25.012 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:25.012 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:24:25.012 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:24:25.231 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:24:25.805 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:47:26 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:24:25.962 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:24:25.962 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:24:25.963 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:24:25.963 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:24:25.963 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:24:25.963 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:24:25.963 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:24:25.963 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:24:25.963 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:24:25.963 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:24:25.963 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:24:25.964 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 52 M 2024-03-25T18:24:25.964 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 308 M 2024-03-25T18:24:25.964 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:25:07.149 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 516 kB/s | 52 MB 01:42 2024-03-25T18:25:07.150 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:07.150 INFO:teuthology.orchestra.run.smithi133.stdout:Total 516 kB/s | 52 MB 01:42 2024-03-25T18:25:07.150 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:25:07.153 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:25:07.154 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:25:07.489 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:25:07.489 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:25:13.418 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:25:13.646 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:25:15.669 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:25:15.933 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:25:15.933 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:25:15.933 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:25:15.933 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:15.933 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:25:15.933 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:25:16.135 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:25:16.701 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 1:48:28 ago on Mon 25 Mar 2024 04:36:48 PM UTC. 2024-03-25T18:25:16.849 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-25T18:25:16.850 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:25:16.850 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-03-25T18:25:16.850 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:25:16.850 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-25T18:25:16.850 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:25:16.850 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:25:16.850 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-25T18:25:16.850 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-03-25T18:25:16.850 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-03-25T18:25:16.851 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:25:16.851 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 1.3 M 2024-03-25T18:25:16.851 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 5.5 M 2024-03-25T18:25:16.851 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-25T18:25:20.616 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 340 kB/s | 1.3 MB 00:03 2024-03-25T18:25:20.617 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:20.617 INFO:teuthology.orchestra.run.smithi133.stdout:Total 340 kB/s | 1.3 MB 00:03 2024-03-25T18:25:20.618 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-25T18:25:20.621 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-25T18:25:20.621 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-25T18:25:20.925 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-25T18:25:21.014 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:21.397 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:21.686 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:21.686 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:25:21.686 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-25T18:25:21.686 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:21.687 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:25:21.687 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-25T18:25:21.882 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:24.952 INFO:teuthology.orchestra.run.smithi055.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 900 kB/s | 52 MB 00:58 2024-03-25T18:25:24.952 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:24.953 INFO:teuthology.orchestra.run.smithi055.stdout:Total 900 kB/s | 52 MB 00:58 2024-03-25T18:25:24.953 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:25:24.957 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:25:24.957 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:25:25.316 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:25:25.317 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:25:31.669 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:25:31.778 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:34.393 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:34.393 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:34.393 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:25:34.393 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:34.394 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:34.394 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:25:34.583 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:25:35.151 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:48:36 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:25:35.306 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:35.307 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 615 k 2024-03-25T18:25:35.308 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 2.1 M 2024-03-25T18:25:35.308 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:25:36.463 INFO:teuthology.orchestra.run.smithi055.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 532 kB/s | 615 kB 00:01 2024-03-25T18:25:36.464 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:36.464 INFO:teuthology.orchestra.run.smithi055.stdout:Total 531 kB/s | 615 kB 00:01 2024-03-25T18:25:36.465 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:25:36.469 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:25:36.469 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:25:36.512 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:25:36.512 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:25:36.686 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:25:36.798 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:25:37.200 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:25:37.478 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:25:37.478 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:37.478 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:25:37.478 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:37.478 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:37.479 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:25:37.665 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:25:38.241 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:48:39 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:25:38.396 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:25:38.397 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:38.397 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:25:38.397 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 771 k 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 2.7 M 2024-03-25T18:25:38.398 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:25:40.117 INFO:teuthology.orchestra.run.smithi055.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 449 kB/s | 771 kB 00:01 2024-03-25T18:25:40.117 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:40.117 INFO:teuthology.orchestra.run.smithi055.stdout:Total 448 kB/s | 771 kB 00:01 2024-03-25T18:25:40.118 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:25:40.122 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:25:40.122 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:25:40.165 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:25:40.165 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:25:40.352 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:25:40.469 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:25:40.961 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:25:41.226 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:25:41.227 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:41.227 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:25:41.227 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:41.227 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:41.227 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:25:41.567 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:25:42.138 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:48:43 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:25:42.293 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:25:42.293 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:25:42.293 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:42.293 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:25:42.293 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:25:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:42.294 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:25:42.294 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:42.294 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:25:42.294 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:42.295 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 1.1 M 2024-03-25T18:25:42.295 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 4.3 M 2024-03-25T18:25:42.295 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:25:43.712 INFO:teuthology.orchestra.run.smithi055.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 812 kB/s | 1.1 MB 00:01 2024-03-25T18:25:43.713 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:43.713 INFO:teuthology.orchestra.run.smithi055.stdout:Total 811 kB/s | 1.1 MB 00:01 2024-03-25T18:25:43.713 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:25:43.718 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:25:43.718 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:25:43.763 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:25:43.763 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:25:44.001 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:25:44.105 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:44.652 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:44.934 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:44.934 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:44.934 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:25:44.934 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:44.934 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:44.935 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:25:45.278 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:25:45.851 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:48:46 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:25:46.007 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:25:46.007 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 337 k 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 1.0 M 2024-03-25T18:25:46.008 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:25:46.932 INFO:teuthology.orchestra.run.smithi055.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 365 kB/s | 337 kB 00:00 2024-03-25T18:25:46.933 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:46.933 INFO:teuthology.orchestra.run.smithi055.stdout:Total 364 kB/s | 337 kB 00:00 2024-03-25T18:25:46.933 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:25:46.938 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:25:46.938 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:25:46.980 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:25:46.980 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:25:47.125 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:25:47.235 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:47.574 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:47.902 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:47.902 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:47.903 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:25:47.903 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:47.903 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:47.903 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:25:48.134 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:25:48.704 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:48:49 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:25:48.861 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:25:48.861 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:48.862 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 509 k 2024-03-25T18:25:48.863 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 2.5 M 2024-03-25T18:25:48.863 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:25:50.261 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 364 kB/s | 509 kB 00:01 2024-03-25T18:25:50.262 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:50.262 INFO:teuthology.orchestra.run.smithi055.stdout:Total 363 kB/s | 509 kB 00:01 2024-03-25T18:25:50.262 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:25:50.266 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:25:50.267 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:25:50.309 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:25:50.310 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:25:50.482 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:25:50.597 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:25:50.988 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:25:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:25:51.309 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:51.309 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:25:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:51.310 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:51.310 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:25:51.511 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:25:52.083 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:48:53 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:25:52.239 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:25:52.240 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:52.240 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:25:52.240 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:52.240 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:25:52.240 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:25:52.240 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:52.241 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:25:52.241 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:25:52.241 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:25:52.241 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:25:52.241 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 52 M 2024-03-25T18:25:52.241 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 298 M 2024-03-25T18:25:52.241 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:26:56.901 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 816 kB/s | 52 MB 01:04 2024-03-25T18:26:56.902 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:26:56.902 INFO:teuthology.orchestra.run.smithi055.stdout:Total 816 kB/s | 52 MB 01:04 2024-03-25T18:26:56.902 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:26:56.905 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:26:56.905 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:26:57.256 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:26:57.257 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:27:03.529 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:27:03.635 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:27:05.724 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:27:06.007 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:27:06.008 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:06.008 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:27:06.008 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:27:06.008 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:06.008 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:27:06.209 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:27:06.786 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 1:50:07 ago on Mon 25 Mar 2024 04:36:59 PM UTC. 2024-03-25T18:27:06.941 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-03-25T18:27:06.941 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:27:06.941 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 1.3 M 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 5.5 M 2024-03-25T18:27:06.942 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-03-25T18:27:09.053 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 607 kB/s | 1.3 MB 00:02 2024-03-25T18:27:09.054 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:27:09.054 INFO:teuthology.orchestra.run.smithi055.stdout:Total 607 kB/s | 1.3 MB 00:02 2024-03-25T18:27:09.054 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-03-25T18:27:09.058 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-03-25T18:27:09.058 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-03-25T18:27:09.105 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-03-25T18:27:09.105 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-03-25T18:27:09.360 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-03-25T18:27:09.463 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:27:09.863 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:27:10.142 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:27:10.142 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:10.142 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-03-25T18:27:10.143 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:27:10.143 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:10.143 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-03-25T18:27:10.346 DEBUG:teuthology.parallel:result is None 2024-03-25T18:27:10.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:27:10.570 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:27:10.590 INFO:teuthology.orchestra.run.smithi055.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:27:10.591 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:27:10.591 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:27:10.593 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:27:10.811 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:27:10.835 INFO:teuthology.orchestra.run.smithi133.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:27:10.835 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:27:10.835 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:27:10.837 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:27:10.837 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:10.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:27:10.867 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:10.867 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:27:10.905 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:27:10.905 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:10.905 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:27:10.933 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:27:11.000 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:11.000 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:27:11.031 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:27:11.102 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:27:11.103 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:11.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:27:11.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:27:11.196 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:11.196 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:27:11.226 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:27:11.296 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:27:11.296 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:11.296 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:27:11.322 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:27:11.388 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:11.388 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:27:11.417 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:27:11.490 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:27:11.576 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:27:11.576 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /var/log/ceph 2024-03-25T18:27:11.579 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /var/log/ceph 2024-03-25T18:27:11.609 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:27:11.609 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:27:11.646 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:27:11.678 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:27:11.678 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:27:11.712 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:27:11.784 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:27:11.784 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:27:11.784 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621283', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi055.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK07BMq4oLcI6wwv4DMUGtscYrh9LL59zIyHfosL9Nj8Ex27kI0DSaI4yISwPX7nrZdnKhC/CUVAG8gzxGw/l/A=', 'smithi133.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqMAS6+p/Y9IuVWSK3DNUKE1ozDOdJ65gM1d/iplZtXgOL9VzTNrOGN+qGVjwh4Pt4gCuRRJXRbtcLOhe0ylkA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:27:11.785 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:27:11.787 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:27:11.803 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:27:11.842 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:27:11.871 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:11.871 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:27:11.921 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:27:11.922 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2024-03-25T18:27:11.976 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:27:11.976 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:27:11.977 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T18:27:11.977 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:27:11.977 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:27:11.977 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-25 16:44:39.737432164 +0000 2024-03-25T18:27:11.977 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-25 16:35:50.085742022 +0000 2024-03-25T18:27:11.977 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-25 16:35:50.085742022 +0000 2024-03-25T18:27:11.977 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-25 16:35:50.085742022 +0000 2024-03-25T18:27:11.977 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:27:12.042 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-25T18:27:12.042 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-25T18:27:12.042 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000318708 s, 1.6 MB/s 2024-03-25T18:27:12.043 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:27:12.099 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2024-03-25T18:27:12.153 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:27:12.154 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:27:12.154 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:27:12.154 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:27:12.154 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:27:12.154 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-25 16:44:39.737432164 +0000 2024-03-25T18:27:12.154 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-25 16:35:50.425735309 +0000 2024-03-25T18:27:12.154 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-25 16:35:50.425735309 +0000 2024-03-25T18:27:12.154 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-25 16:35:50.425735309 +0000 2024-03-25T18:27:12.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:27:12.219 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-25T18:27:12.219 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-25T18:27:12.219 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000248743 s, 2.1 MB/s 2024-03-25T18:27:12.220 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:27:12.275 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2024-03-25T18:27:12.330 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:27:12.330 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:27:12.330 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T18:27:12.330 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:27:12.330 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:27:12.330 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-25 16:44:39.738432144 +0000 2024-03-25T18:27:12.330 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-25 16:35:50.778728340 +0000 2024-03-25T18:27:12.330 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-25 16:35:50.778728340 +0000 2024-03-25T18:27:12.330 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-25 16:35:50.778728340 +0000 2024-03-25T18:27:12.330 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:27:12.394 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-25T18:27:12.395 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-25T18:27:12.395 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.00042149 s, 1.2 MB/s 2024-03-25T18:27:12.396 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:27:12.450 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2024-03-25T18:27:12.505 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:27:12.505 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:27:12.505 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:27:12.505 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:27:12.505 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:27:12.505 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-03-25 16:44:39.738432144 +0000 2024-03-25T18:27:12.505 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-03-25 16:35:51.143721134 +0000 2024-03-25T18:27:12.505 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-03-25 16:35:51.143721134 +0000 2024-03-25T18:27:12.506 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-03-25 16:35:51.143721134 +0000 2024-03-25T18:27:12.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:27:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-03-25T18:27:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-03-25T18:27:12.572 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000452163 s, 1.1 MB/s 2024-03-25T18:27:12.573 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:27:12.629 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:12.629 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:12.629 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:27:12.647 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:27:12.648 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 2024-03-25T18:27:12.705 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:27:12.706 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:27:12.706 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:27:12.706 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:27:12.706 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:27:12.706 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-03-25 16:45:49.521066812 +0000 2024-03-25T18:27:12.706 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-03-25 16:35:46.939442892 +0000 2024-03-25T18:27:12.706 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-03-25 16:35:46.939442892 +0000 2024-03-25T18:27:12.706 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-03-25 16:35:46.939442892 +0000 2024-03-25T18:27:12.706 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:27:12.775 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-03-25T18:27:12.775 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-03-25T18:27:12.775 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000302272 s, 1.7 MB/s 2024-03-25T18:27:12.777 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:27:12.835 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 2024-03-25T18:27:12.891 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:27:12.891 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:27:12.891 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T18:27:12.891 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:27:12.891 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:27:12.891 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-03-25 16:45:49.522066793 +0000 2024-03-25T18:27:12.892 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-03-25 16:35:47.286436264 +0000 2024-03-25T18:27:12.892 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-03-25 16:35:47.286436264 +0000 2024-03-25T18:27:12.892 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-03-25 16:35:47.286436264 +0000 2024-03-25T18:27:12.892 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:27:12.958 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-03-25T18:27:12.958 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-03-25T18:27:12.959 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000240648 s, 2.1 MB/s 2024-03-25T18:27:12.960 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:27:13.017 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 2024-03-25T18:27:13.072 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:27:13.072 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:27:13.072 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:27:13.072 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:27:13.073 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:27:13.073 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-03-25 16:45:49.522066793 +0000 2024-03-25T18:27:13.073 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-03-25 16:35:47.627429751 +0000 2024-03-25T18:27:13.073 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-03-25 16:35:47.627429751 +0000 2024-03-25T18:27:13.073 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-03-25 16:35:47.627429751 +0000 2024-03-25T18:27:13.073 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:27:13.138 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-03-25T18:27:13.138 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-03-25T18:27:13.138 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000232827 s, 2.2 MB/s 2024-03-25T18:27:13.139 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:27:13.195 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 2024-03-25T18:27:13.250 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:27:13.250 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:27:13.250 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:27:13.250 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:27:13.250 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:27:13.250 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-03-25 16:45:49.522066793 +0000 2024-03-25T18:27:13.250 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-03-25 16:35:47.975423104 +0000 2024-03-25T18:27:13.250 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-03-25 16:35:47.975423104 +0000 2024-03-25T18:27:13.250 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-03-25 16:35:47.975423104 +0000 2024-03-25T18:27:13.251 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:27:13.315 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-03-25T18:27:13.315 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-03-25T18:27:13.316 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000218393 s, 2.3 MB/s 2024-03-25T18:27:13.316 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:27:13.373 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:13.373 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:27:13.373 INFO:tasks.ceph:Generating config... 2024-03-25T18:27:13.374 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:27:13.374 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:27:13.374 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:27:13.375 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:27:13.375 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:27:13.375 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:27:13.375 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:27:13.375 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:27:13.375 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:27:13.375 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:27:13.375 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:27:13.375 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:27:13.375 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:27:13.375 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:27:13.375 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:27:13.375 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:27:13.376 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:27:13.376 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:27:13.376 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:27:13.376 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:27:13.376 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:27:13.376 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:27:13.376 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:27:13.376 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:27:13.376 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:27:13.376 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:27:13.376 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:27:13.376 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:27:13.377 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:27:13.378 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:27:13.379 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:27:13.379 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:27:13.379 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:27:13.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:27:13.439 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:27:13.442 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:27:13.491 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:27:13.518 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.55:6789]'), ('mon.c', '[v1:172.21.15.55:6790]'), ('mon.b', '[v1:172.21.15.133:6789]')] 2024-03-25T18:27:13.518 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.55:6789],[v1:172.21.15.55:6790],[v1:172.21.15.133:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:27:13.519 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:13.519 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:27:13.573 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.55:6789]' --addv c '[v1:172.21.15.55:6790]' --addv b '[v1:172.21.15.133:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:13.657 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:13.657 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: generated fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:27:13.657 INFO:teuthology.orchestra.run.smithi055.stdout:setting min_mon_release = quincy 2024-03-25T18:27:13.657 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 0 2024-03-25T18:27:13.657 INFO:teuthology.orchestra.run.smithi055.stdout:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:27:13.657 INFO:teuthology.orchestra.run.smithi055.stdout:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:27:13.657 INFO:teuthology.orchestra.run.smithi055.stdout:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:27:13.657 INFO:teuthology.orchestra.run.smithi055.stdout:min_mon_release 17 (quincy) 2024-03-25T18:27:13.658 INFO:teuthology.orchestra.run.smithi055.stdout:election_strategy: 3 2024-03-25T18:27:13.658 INFO:teuthology.orchestra.run.smithi055.stdout:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:27:13.658 INFO:teuthology.orchestra.run.smithi055.stdout:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:27:13.658 INFO:teuthology.orchestra.run.smithi055.stdout:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:27:13.658 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:27:13.659 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:27:13.673 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a9b133ab-c902-44c5-bfee-b6d6f33f1d56... 2024-03-25T18:27:13.674 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:27:13.716 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: chdir = "" 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: auth supported = cephx 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: filestore xattr use omap = true 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: mon clock drift allowed = 1.000 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.763 INFO:teuthology.orchestra.run.smithi055.stdout: osd crush chooseleaf type = 0 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: auth debug = true 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on old message = true 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on bug = true 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: mon pg warn max object skew = 0 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default size = 2 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow primary affinity = true 2024-03-25T18:27:13.764 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow pg remap = true 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on no sortbitwise = false 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on too few osds = false 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: mon_allow_pool_size_one = true 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: osd default data pool replay window = 5 2024-03-25T18:27:13.765 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: mon allow pool delete = true 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: mon cluster log file level = debug 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: debug asserts on shutdown = true 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: mon health detail to clog = false 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: mon host = "[v1:172.21.15.55:6789],[v1:172.21.15.55:6790],[v1:172.21.15.133:6789]" 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: mon client directed command retry = 5 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: mon election default strategy = 3 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: ms bind msgr2 = False 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: ms inject socket failures = 5000 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: ms type = async 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: osd heartbeat grace = 80 2024-03-25T18:27:13.766 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout:[osd] 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd journal size = 100 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub load threshold = 5.0 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub max interval = 600 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd recover clone overlap = true 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd recovery max chunk = 1048576 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug shutdown = true 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug op order = true 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify stray on activate = true 2024-03-25T18:27:13.767 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug trim objects = true 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: osd open classes on start = true 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug pg log writeout = true 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: osd map max advance = 10 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: journal zero on create = true 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: filestore apply finisher threads = 3 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: bdev debug aio = true 2024-03-25T18:27:13.768 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug misdirected ops = true 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: bdev async discard = True 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: bdev enable discard = True 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore allocator = stupid 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore block size = 96636764160 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore fsck on mount = True 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore zero block detection = True 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluefs = 1 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluestore = 1 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: debug monc = 20 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: debug osd = 20 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: debug rocksdb = 10 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd full ratio = 0.9 2024-03-25T18:27:13.769 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd class default list = * 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd class load list = * 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd client watch timeout = 120 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify cached snaps = True 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify missing on start = True 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd fast shutdown = False 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd max markdown count = 1000 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd mclock override recovery settings = True 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd objectstore = bluestore 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd op queue = debug_random 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd op queue cut off = debug_random 2024-03-25T18:27:13.770 INFO:teuthology.orchestra.run.smithi055.stdout: osd shutdown pgref assert = True 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr] 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: debug mgr = 20 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: mgr/telemetry/nag = false 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout:[mon] 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: debug paxos = 20 2024-03-25T18:27:13.771 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: mon data avail warn = 5 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: mon mgr mkfs grace = 240 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd reporter subtree level = osd 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd prime pg temp = true 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.772 INFO:teuthology.orchestra.run.smithi055.stdout: # 1m isn't quite enough 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: mon min osdmap epochs = 50 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd crush smoke test = False 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: mon osdmap full prune interval = 2 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: mon osdmap full prune min = 15 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: mon scrub interval = 300 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on pool no app = False 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: paxos service trim min = 10 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout:[client] 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: rgw cache enabled = true 2024-03-25T18:27:13.773 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable ops log = true 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable usage log = true 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout: debug monc = 20 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout: debug objecter = 20 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout: debug rados = 20 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.a] 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.c] 2024-03-25T18:27:13.774 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.b] 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2024-03-25T18:27:13.775 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2024-03-25T18:27:13.776 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: mon_allow_pool_size_one = true 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.777 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: mon health detail to clog = false 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = "[v1:172.21.15.55:6789],[v1:172.21.15.55:6790],[v1:172.21.15.133:6789]" 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: mon client directed command retry = 5 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: mon election default strategy = 3 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: ms bind msgr2 = False 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: ms inject socket failures = 5000 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: ms type = async 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: osd heartbeat grace = 80 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2024-03-25T18:27:13.778 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug trim objects = true 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2024-03-25T18:27:13.779 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: bdev async discard = True 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: bdev enable discard = True 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore allocator = stupid 2024-03-25T18:27:13.780 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore block size = 96636764160 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore fsck on mount = True 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: bluestore zero block detection = True 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: debug bluefs = 1 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: debug bluestore = 1 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: debug monc = 20 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 20 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: debug rocksdb = 10 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd full ratio = 0.9 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: osd class default list = * 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: osd class load list = * 2024-03-25T18:27:13.781 INFO:teuthology.orchestra.run.smithi133.stdout: osd client watch timeout = 120 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify cached snaps = True 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify missing on start = True 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd fast shutdown = False 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd max markdown count = 1000 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd mclock override recovery settings = True 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd objectstore = bluestore 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd op queue = debug_random 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd op queue cut off = debug_random 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: osd shutdown pgref assert = True 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2024-03-25T18:27:13.782 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:27:13.783 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: # 1m isn't quite enough 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:27:13.784 INFO:teuthology.orchestra.run.smithi133.stdout: mon min osdmap epochs = 50 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd crush smoke test = False 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: mon osdmap full prune interval = 2 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: mon osdmap full prune min = 15 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: mon scrub interval = 300 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on pool no app = False 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: paxos service trim min = 10 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:27:13.785 INFO:teuthology.orchestra.run.smithi133.stdout: debug monc = 20 2024-03-25T18:27:13.786 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2024-03-25T18:27:13.786 INFO:teuthology.orchestra.run.smithi133.stdout: debug objecter = 20 2024-03-25T18:27:13.786 INFO:teuthology.orchestra.run.smithi133.stdout: debug rados = 20 2024-03-25T18:27:13.786 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2024-03-25T18:27:13.786 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.c] 2024-03-25T18:27:13.786 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.b] 2024-03-25T18:27:13.787 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:27:13.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:27:13.842 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:27:13.843 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:13.843 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:27:13.857 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:13.857 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:27:13.911 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:27:13.911 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:13.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:27:13.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:27:13.989 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:13.989 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:14.044 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:27:14.044 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:14.044 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:27:14.044 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:27:14.083 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:14.083 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:14.138 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:27:14.138 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:27:14.138 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:27:14.192 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:27:14.195 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:27:14.278 INFO:teuthology.orchestra.run.smithi133.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:27:14.280 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:27:14.281 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:27:14.281 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:27:14.322 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:27:14.338 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:27:14.380 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:27:14.395 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:27:14.395 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:27:14.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:27:14.422 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:14.422 INFO:tasks.ceph:role: osd.0 2024-03-25T18:27:14.422 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:27:14.422 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:27:14.516 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:27:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:27:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:27:14.516 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:27:14.517 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:27:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:27:14.517 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:27:14.517 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:27:14.517 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:27:14.517 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:27:15.836 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-25T18:27:15.838 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-25T18:27:15.838 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:27:15.876 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:27:15.950 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:27:16.019 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:16.019 INFO:tasks.ceph:role: osd.1 2024-03-25T18:27:16.019 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:27:16.020 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:27:16.118 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:27:16.118 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:27:16.118 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:27:16.118 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:27:16.118 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:27:16.118 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:27:16.118 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:27:16.119 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:27:16.119 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:27:16.119 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:27:17.439 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-25T18:27:17.439 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-25T18:27:17.439 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:27:17.478 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:27:17.553 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:27:17.622 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:17.622 INFO:tasks.ceph:role: osd.2 2024-03-25T18:27:17.622 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:27:17.622 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:27:17.721 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:27:19.039 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-25T18:27:19.041 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-25T18:27:19.042 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:27:19.079 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:27:19.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:27:19.221 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:19.222 INFO:tasks.ceph:role: osd.3 2024-03-25T18:27:19.222 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:27:19.222 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:27:19.320 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:27:19.320 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:27:19.320 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:27:19.320 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:27:19.320 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:27:19.320 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:27:19.321 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:27:19.321 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:27:19.321 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:27:19.321 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:27:20.663 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2024-03-25T18:27:20.666 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2024-03-25T18:27:20.666 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:27:20.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:27:20.776 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:20.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:20.883+0000 7f9958d38640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:27:20.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:20.884+0000 7f9958d38640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:27:20.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:20.884+0000 7f9958d38640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:27:20.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:20.884+0000 7f9958d38640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:27:20.885 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:20.884+0000 7f9958d38640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:27:20.886 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:20.885+0000 7f9958d38640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:27:23.753 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:27:23.784 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:23.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:23.875+0000 7f7af5013640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:27:23.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:23.875+0000 7f7af5013640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:27:23.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:23.875+0000 7f7af5013640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:27:23.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:23.875+0000 7f7af5013640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:27:23.876 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:23.875+0000 7f7af5013640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:27:23.877 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:23.876+0000 7f7af5013640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:27:26.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:27:26.766 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:26.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:26.856+0000 7f5834811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:27:26.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:26.856+0000 7f5834811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:27:26.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:26.856+0000 7f5834811640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:27:26.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:26.856+0000 7f5834811640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:27:26.858 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:26.856+0000 7f5834811640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:27:26.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:26.857+0000 7f5834811640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:27:29.718 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:27:29.750 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:29.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:29.840+0000 7fa703048640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:27:29.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:29.840+0000 7fa703048640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:27:29.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:29.840+0000 7fa703048640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:27:29.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:29.840+0000 7fa703048640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:27:29.841 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:29.840+0000 7fa703048640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:27:29.842 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:29.841+0000 7fa703048640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:27:32.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:27:32.734 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:27:32.765 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:32.765 INFO:tasks.ceph:role: osd.4 2024-03-25T18:27:32.765 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:27:32.766 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:27:32.843 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:27:33.354 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2024-03-25T18:27:33.356 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2024-03-25T18:27:33.357 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:27:33.490 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:27:33.526 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:27:33.595 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:33.595 INFO:tasks.ceph:role: osd.5 2024-03-25T18:27:33.595 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:27:33.595 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:27:33.670 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:27:33.671 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:27:33.671 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:27:33.671 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:27:33.671 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:27:33.671 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:27:33.671 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:27:33.671 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:27:33.671 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:27:33.671 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:27:34.121 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2024-03-25T18:27:34.123 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2024-03-25T18:27:34.123 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:27:34.211 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:27:34.243 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:27:34.310 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:34.310 INFO:tasks.ceph:role: osd.6 2024-03-25T18:27:34.310 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:27:34.311 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:27:34.384 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:27:34.385 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:27:34.385 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:27:34.385 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:27:34.385 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:27:34.385 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:27:34.385 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:27:34.385 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:27:34.385 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:27:34.385 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:27:34.878 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2024-03-25T18:27:34.879 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2024-03-25T18:27:34.879 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:27:35.010 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:27:35.042 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:27:35.109 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:27:35.109 INFO:tasks.ceph:role: osd.7 2024-03-25T18:27:35.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:27:35.109 DEBUG:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:27:35.183 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:27:35.184 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:27:35.184 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:27:35.184 INFO:teuthology.orchestra.run.smithi133.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:27:35.184 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:27:35.184 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:27:35.184 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:27:35.184 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:27:35.184 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:27:35.184 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:27:35.698 INFO:teuthology.orchestra.run.smithi133.stdout:Discarding blocks...Done. 2024-03-25T18:27:35.700 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2024-03-25T18:27:35.700 DEBUG:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:27:35.819 DEBUG:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:27:35.851 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:35.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:35.950+0000 7f34a6b0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:27:35.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:35.950+0000 7f34a6b0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:27:35.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:35.950+0000 7f34a6b0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:27:35.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:35.950+0000 7f34a6b0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:27:35.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:35.950+0000 7f34a6b0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:27:35.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:35.951+0000 7f34a6b0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:27:38.791 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:27:38.821 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:38.914 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:38.913+0000 7f3c34887640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:27:38.914 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:38.913+0000 7f3c34887640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:27:38.914 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:38.913+0000 7f3c34887640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:27:38.914 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:38.913+0000 7f3c34887640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:27:38.914 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:38.913+0000 7f3c34887640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:27:38.914 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:38.914+0000 7f3c34887640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:27:41.758 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:27:41.789 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:41.883 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:41.882+0000 7fe3219ff640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:27:41.883 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:41.882+0000 7fe3219ff640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:27:41.883 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:41.882+0000 7fe3219ff640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:27:41.883 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:41.882+0000 7fe3219ff640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:27:41.883 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:41.882+0000 7fe3219ff640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:27:41.883 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:41.883+0000 7fe3219ff640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:27:44.725 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:27:44.755 DEBUG:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:44.846 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:44.845+0000 7f419760c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:27:44.846 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:44.845+0000 7f419760c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:27:44.846 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:44.845+0000 7f419760c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:27:44.846 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:44.845+0000 7f419760c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:27:44.846 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:44.845+0000 7f419760c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:27:44.847 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:27:44.846+0000 7f419760c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:27:47.692 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:27:47.723 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:27:47.723 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:47.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:27:47.755 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:47.755 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:27:47.825 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:47.826 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:27:47.895 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:47.895 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:27:47.965 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:47.965 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:27:48.033 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:48.033 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:27:48.065 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:48.065 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:27:48.136 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:48.136 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:27:48.206 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:48.206 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:27:48.275 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:48.275 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:27:48.344 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:27:48.344 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:27:48.361 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:27:48.361 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:27:48.400 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:27:48.400 DEBUG:teuthology.orchestra.run.smithi055:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:27:48.403 DEBUG:teuthology.orchestra.run.smithi133:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:27:48.443 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.y] 2024-03-25T18:27:48.443 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQACwgFmcbJ9CxAA1Ff1/qVcl5tHH/3jNztnzA== 2024-03-25T18:27:48.443 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.0] 2024-03-25T18:27:48.443 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAIwgFmRwK9NBAAZKp9dpT2lE7Dc2KJ699MZw== 2024-03-25T18:27:48.443 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.1] 2024-03-25T18:27:48.443 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQALwgFmZME2NBAAUZXDpjrNUTL1LkVg2ljpcw== 2024-03-25T18:27:48.443 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.2] 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAOwgFmmzIYMxAAGdJcRm9v/mqaas/4Tnvnxg== 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.3] 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQARwgFmJvIhMhAABrUsF0NyDuZfnFrgsDPiyg== 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.x] 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQACwgFmv+GaEBAAQ/tkqhPTINpZq38/W6bjCg== 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.4] 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAXwgFmzOSsOBAAG5Bk0UJk1MbyYoLMPvEJMw== 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.5] 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAawgFmjDl4NhAA1UXDyf3re35lGg4hnSU6aA== 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.6] 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAdwgFmNzWbNBAAF+NeDe/BDNYa/NTjZs7o8w== 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.7] 2024-03-25T18:27:48.444 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQAgwgFmRRRpMhAAbtBxXs4LfU3seRFYqn6Z3g== 2024-03-25T18:27:48.445 INFO:teuthology.orchestra.run.smithi055.stdout:[client.0] 2024-03-25T18:27:48.445 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQACwgFm1o5EExAAJvOT0PJLlx+1sOhsw6uS5w== 2024-03-25T18:27:48.445 INFO:teuthology.orchestra.run.smithi055.stdout:[client.1] 2024-03-25T18:27:48.445 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQACwgFmrVupFhAAf8Ih3hDB9eytaECMbXEVhQ== 2024-03-25T18:27:48.466 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr.y] 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQACwgFmcbJ9CxAA1Ff1/qVcl5tHH/3jNztnzA== 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAIwgFmRwK9NBAAZKp9dpT2lE7Dc2KJ699MZw== 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQALwgFmZME2NBAAUZXDpjrNUTL1LkVg2ljpcw== 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAOwgFmmzIYMxAAGdJcRm9v/mqaas/4Tnvnxg== 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.3] 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQARwgFmJvIhMhAABrUsF0NyDuZfnFrgsDPiyg== 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr.x] 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQACwgFmv+GaEBAAQ/tkqhPTINpZq38/W6bjCg== 2024-03-25T18:27:48.467 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.4] 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAXwgFmzOSsOBAAG5Bk0UJk1MbyYoLMPvEJMw== 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.5] 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAawgFmjDl4NhAA1UXDyf3re35lGg4hnSU6aA== 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.6] 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAdwgFmNzWbNBAAF+NeDe/BDNYa/NTjZs7o8w== 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.7] 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAgwgFmRRRpMhAAbtBxXs4LfU3seRFYqn6Z3g== 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQACwgFm1o5EExAAJvOT0PJLlx+1sOhsw6uS5w== 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout:[client.1] 2024-03-25T18:27:48.468 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQACwgFmrVupFhAAf8Ih3hDB9eytaECMbXEVhQ== 2024-03-25T18:27:48.469 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:27:48.488 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:27:48.563 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.565 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.618 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.620 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.671 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.673 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.726 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.777 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:27:48.780 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:27:48.830 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.833 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.884 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.886 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.938 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.940 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.992 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:48.994 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:27:49.045 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:27:49.047 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:27:49.101 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:27:49.103 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:27:49.157 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:27:49.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:27:49.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:27:49.280 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:27:49.306 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:27:49.373 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:27:49.474 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:27:49.501 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:27:49.529 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:27:49.628 DEBUG:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:27:49.654 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:49.657 DEBUG:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:27:49.709 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:27:49.710 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:27:49.710 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:27:49.710 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:27:49.712 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:27:49.712 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:27:49.713 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:27:49.713 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:27:49.714 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:27:49.715 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:27:49.715 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:27:49.715 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:27:49.753 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:27:49.753 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:27:49.753 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:27:49.753 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:27:49.756 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:27:49.756 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:27:49.756 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:27:49.758 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:27:49.758 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:27:49.758 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:27:49.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.859+0000 7f2add8f6640 10 monclient: get_monmap_and_config 2024-03-25T18:27:49.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.859+0000 7f2add8f6640 10 monclient: build_initial_monmap 2024-03-25T18:27:49.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.859+0000 7f2add8f6640 10 monclient: monmap: 2024-03-25T18:27:49.860 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 0 2024-03-25T18:27:49.860 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:27:49.860 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:27:49.860413+0000 2024-03-25T18:27:49.861 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:27:49.860413+0000 2024-03-25T18:27:49.861 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:49.861 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:27:49.861 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:27:49.861 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:27:49.861 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:27:49.861 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:27:49.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.860+0000 7f2add8f6640 1 Processor -- start 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.860+0000 7f2add8f6640 1 -- start start 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.860+0000 7f2add8f6640 10 monclient: init 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.860+0000 7f2add8f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.860+0000 7f2add8f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.860+0000 7f2add8f6640 10 monclient(hunting): picked mon.noname-b con 0x7f2ad80b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.860+0000 7f2add8f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad8142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.861+0000 7f2add8f6640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.861+0000 7f2add8f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80a9d60 con 0x7f2ad80a96b0 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.861+0000 7f2add8f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad813ff10 con 0x7f2ad8142a20 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.861+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ad8142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.861+0000 7f2add8f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80aa1f0 con 0x7f2ad80b4700 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.861+0000 7f2ad77fe640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ad813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.861+0000 7f2add8f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.861+0000 7f2add8f6640 10 monclient(hunting): authenticate will time out at 7274.118443s 2024-03-25T18:27:49.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:49.861+0000 7f2acffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ad813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:50.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:50.062+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ad8142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:50.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:50.062+0000 7f2ad77fe640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ad813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:50.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:50.062+0000 7f2acffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ad813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:50.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:50.463+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ad8142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:50.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:50.463+0000 7f2ad77fe640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ad813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:50.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:50.463+0000 7f2acffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ad813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:51.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:51.264+0000 7f2ad77fe640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ad813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:51.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:51.264+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ad8142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:51.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:51.264+0000 7f2acffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ad813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:52.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 10 monclient(hunting): tick 2024-03-25T18:27:52.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ad8142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ad813f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ad813f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2ad80b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad8142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80b4ae0 con 0x7f2ad80a96b0 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80a9d60 con 0x7f2ad8142a20 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad813ff10 con 0x7f2ad80b4700 2024-03-25T18:27:52.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad57fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:52.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2acffff640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:52.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:52.861+0000 7f2ad77fe640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab800c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:53.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:53.062+0000 7f2acffff640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:53.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:53.062+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:53.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:53.062+0000 7f2ad77fe640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab800c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:53.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:53.463+0000 7f2acffff640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:53.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:53.463+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:53.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:53.463+0000 7f2ad77fe640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab800c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:54.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:54.264+0000 7f2acffff640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:54.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:54.264+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:54.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:54.264+0000 7f2ad77fe640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab800c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:55.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 10 monclient(hunting): tick 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab800c730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2ad80b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad8142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80aa1f0 con 0x7f2ad80a96b0 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80b4ae0 con 0x7f2ad8142a20 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80a9d60 con 0x7f2ad80b4700 2024-03-25T18:27:55.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad57fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:55.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:55.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2ad77fe640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80106b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:55.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:55.861+0000 7f2acffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:56.062 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:56.062+0000 7f2ad77fe640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80106b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:56.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:56.062+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:56.063 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:56.062+0000 7f2acffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:56.463 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:56.463+0000 7f2ad77fe640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80106b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:56.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:56.463+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:56.464 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:56.463+0000 7f2acffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:56.715 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:00:06.852 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:57.021 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:00:07.159 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:57.224 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:00:07.394 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:57.230 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:00:07.400 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:57.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:57.264+0000 7f2ad77fe640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80106b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6789/0 2024-03-25T18:27:57.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:57.264+0000 7f2ad6ffd640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.55:6790/0 2024-03-25T18:27:57.265 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:57.264+0000 7f2acffff640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.133:6789/0 2024-03-25T18:27:57.551 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:00:07.721 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:57.557 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:00:07.727 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 10 monclient(hunting): tick 2024-03-25T18:27:58.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:27:58.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:27:58.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 1 -- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 1 -- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 1 -- >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80106b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad8142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2ad80b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad813ff10 con 0x7f2ad80a96b0 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80aa1f0 con 0x7f2ad8142a20 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80b4ae0 con 0x7f2ad80b4700 2024-03-25T18:27:58.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.861+0000 7f2ad57fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:58.919 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.919+0000 7f2acffff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 0x7f2ab8008ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:44358/0 (socket says 172.21.15.55:44358) 2024-03-25T18:27:58.920 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:58.919+0000 7f2acffff640 1 -- 172.21.15.55:0/1465675037 learned_addr learned my addr 172.21.15.55:0/1465675037 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:27:59.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:59.260+0000 7f2ad77fe640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T18:27:59.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:59.260+0000 7f2ad77fe640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:27:59.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:59.260+0000 7f2ad77fe640 1 --1- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 0x7f2ab800ffa0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:27:59.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:59.265+0000 7f2ad6ffd640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T18:27:59.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:59.265+0000 7f2ad6ffd640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:27:59.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:59.265+0000 7f2ad6ffd640 1 --1- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 0x7f2ab80056b0 :-1 s=OPENED pgs=3 cs=1 l=1).handle_message read tag failed 2024-03-25T18:27:59.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:59.265+0000 7f2ad6ffd640 1 --1- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 0x7f2ab80056b0 :-1 s=OPENED pgs=3 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:27:59.266 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:27:59.265+0000 7f2ad67fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.55:6789/0 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 10 monclient(hunting): tick 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6790/0 conn(0x7f2ad80b4700 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab8008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80056b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2ab80106b0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:01.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:01.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad8142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:01.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80a9d60 con 0x7f2ad80a96b0 2024-03-25T18:28:01.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab8010b30 con 0x7f2ad8142a20 2024-03-25T18:28:01.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab8007ea0 con 0x7f2ab80106b0 2024-03-25T18:28:01.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:01.861+0000 7f2ad57fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:04.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 10 monclient(hunting): tick 2024-03-25T18:28:04.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:28:04.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:28:04.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:28:04.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6790/0 conn(0x7f2ab80106b0 legacy=0x7f2ab800d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2ab8008030 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad8142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab80090e0 con 0x7f2ad80a96b0 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab80093b0 con 0x7f2ad8142a20 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab8008b70 con 0x7f2ab8008030 2024-03-25T18:28:04.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:04.862+0000 7f2ad57fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:07.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 10 monclient(hunting): tick 2024-03-25T18:28:07.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6790/0 conn(0x7f2ab8008030 legacy=0x7f2ab8008410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.133:6789/0 conn(0x7f2ad8142a20 legacy=0x7f2ab800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab800d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2ab80056b0 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2ab800ffa0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab8008dc0 con 0x7f2ad80a96b0 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab80104c0 con 0x7f2ab80056b0 2024-03-25T18:28:07.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab8006a20 con 0x7f2ab800ffa0 2024-03-25T18:28:07.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:07.862+0000 7f2ad57fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 10 monclient(hunting): tick 2024-03-25T18:28:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:28:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:28:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:28:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6790/0 conn(0x7f2ab800ffa0 legacy=0x7f2ab8008410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.133:6789/0 conn(0x7f2ab80056b0 legacy=0x7f2ab8005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.864 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:10.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2ab800a700 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:10.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2ab800ffa0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:10.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:10.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab8006c50 con 0x7f2ad80a96b0 2024-03-25T18:28:10.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab8005b30 con 0x7f2ab800a700 2024-03-25T18:28:10.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab8005d60 con 0x7f2ab800ffa0 2024-03-25T18:28:10.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:10.863+0000 7f2ad57fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:13.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.285+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029818489 0 0) 0x7f2ab8006c50 con 0x7f2ad80a96b0 2024-03-25T18:28:13.286 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.285+0000 7f2ad67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:13.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.285+0000 7f2ad67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:13.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.285+0000 7f2ad67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:13.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.285+0000 7f2ad67fc640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:28:13.287 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.285+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac4003520 con 0x7f2ad80a96b0 2024-03-25T18:28:13.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.314+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875593430 0 0) 0x7f2ab8005b30 con 0x7f2ab800a700 2024-03-25T18:28:13.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.314+0000 7f2ad67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:13.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.314+0000 7f2ad67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:13.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.314+0000 7f2ad67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:13.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.314+0000 7f2ad67fc640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:28:13.315 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.314+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab8006c50 con 0x7f2ab800a700 2024-03-25T18:28:13.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.329+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306458642 0 0) 0x7f2ab8005d60 con 0x7f2ab800ffa0 2024-03-25T18:28:13.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.329+0000 7f2ad67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:13.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.329+0000 7f2ad67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:13.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.329+0000 7f2ad67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:13.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.329+0000 7f2ad67fc640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:28:13.330 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.329+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab8005b30 con 0x7f2ab800ffa0 2024-03-25T18:28:13.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2453624283 0 0) 0x7f2ac4003520 con 0x7f2ad80a96b0 2024-03-25T18:28:13.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6790/0 conn(0x7f2ab800ffa0 legacy=0x7f2ab8008410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:13.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.133:6789/0 conn(0x7f2ab800a700 legacy=0x7f2ab800cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:13.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:13.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 1 monclient: found mon.noname-a 2024-03-25T18:28:13.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:13.334 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab8005f90 con 0x7f2ad80a96b0 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2ad0004220 con 0x7f2ad80a96b0 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2add8f6640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2add8f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2ad67fc640 10 monclient: dump: 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:13.335 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:13.336 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:13.336 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:13.336 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:13.336 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:13.336 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.333+0000 7f2add8f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:13.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.373+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ad0004380 con 0x7f2ad80a96b0 2024-03-25T18:28:13.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.373+0000 7f2ad67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:13.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.373+0000 7f2ad67fc640 1 -- 172.21.15.55:0/1465675037 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2ad00047d0 con 0x7f2ad80a96b0 2024-03-25T18:28:13.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.373+0000 7f2ad67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:13.374 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.373+0000 7f2ad67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.373+0000 7f2ad67fc640 10 monclient: dump: 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.373+0000 7f2add8f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:13.375 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.373+0000 7f2add8f6640 4 set_mon_vals no callback set 2024-03-25T18:28:13.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.374+0000 7f2add8f6640 10 monclient: shutdown 2024-03-25T18:28:13.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.374+0000 7f2add8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:13.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.374+0000 7f2add8f6640 1 -- 172.21.15.55:0/1465675037 >> v1:172.21.15.55:6789/0 conn(0x7f2ad80a96b0 legacy=0x7f2ab80062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:13.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.374+0000 7f2add8f6640 1 -- 172.21.15.55:0/1465675037 shutdown_connections 2024-03-25T18:28:13.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.374+0000 7f2add8f6640 1 -- 172.21.15.55:0/1465675037 >> 172.21.15.55:0/1465675037 conn(0x7f2ad80668e0 msgr2=0x7f2ad8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:13.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.374+0000 7f2add8f6640 1 -- 172.21.15.55:0/1465675037 shutdown_connections 2024-03-25T18:28:13.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.374+0000 7f2add8f6640 1 -- 172.21.15.55:0/1465675037 wait complete. 2024-03-25T18:28:13.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.375+0000 7f2add8f6640 10 monclient: build_initial_monmap 2024-03-25T18:28:13.378 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.375+0000 7f2add8f6640 10 monclient: monmap: 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 0 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 0.000000 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:created 0.000000 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.375+0000 7f2add8f6640 1 librados: starting msgr at 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.375+0000 7f2add8f6640 1 librados: starting objecter 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.375+0000 7f2add8f6640 1 Processor -- start 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.375+0000 7f2add8f6640 1 -- start start 2024-03-25T18:28:13.379 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.375+0000 7f2add8f6640 1 librados: setting wanted keys 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.375+0000 7f2add8f6640 1 librados: calling monclient init 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.375+0000 7f2add8f6640 10 monclient: init 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 10 monclient(hunting): picked mon.noname-b con 0x7f2ab800a700 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 10 monclient(hunting): picked mon.noname-a con 0x7f2ab800ffa0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad80a96b0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad8120ad0 con 0x7f2ab800ffa0 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad8139f90 con 0x7f2ab800a700 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad813ace0 con 0x7f2ad80a96b0 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2add8f6640 10 monclient(hunting): authenticate will time out at 7297.633587s 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2ad77fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f2ad80a96b0 0x7f2ad8139880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:49770/0 (socket says 172.21.15.55:49770) 2024-03-25T18:28:13.380 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2ad77fe640 1 -- 172.21.15.55:0/593489309 learned_addr learned my addr 172.21.15.55:0/593489309 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:13.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.376+0000 7f2ad6ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2ab800a700 0x7f2ad80b0400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:49746/0 (socket says 172.21.15.55:49746) 2024-03-25T18:28:13.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.377+0000 7f2ad4ff9640 10 client.?.objecter ms_handle_connect 0x7f2ab800a700 2024-03-25T18:28:13.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.377+0000 7f2ad4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:13.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.377+0000 7f2ad4ff9640 10 client.?.objecter ms_handle_connect 0x7f2ad80a96b0 2024-03-25T18:28:13.381 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.377+0000 7f2ad4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:13.426 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.426+0000 7f2ad4ff9640 10 client.?.objecter ms_handle_connect 0x7f2ab800ffa0 2024-03-25T18:28:13.427 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.426+0000 7f2ad4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:13.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.432+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134702448 0 0) 0x7f2ad8120ad0 con 0x7f2ab800ffa0 2024-03-25T18:28:13.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.432+0000 7f2ad4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:13.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.432+0000 7f2ad4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:13.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.432+0000 7f2ad4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:13.433 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.432+0000 7f2ad4ff9640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:28:13.434 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.432+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab8005d60 con 0x7f2ab800ffa0 2024-03-25T18:28:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.445+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 826462167 0 0) 0x7f2ab8005d60 con 0x7f2ab800ffa0 2024-03-25T18:28:13.446 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.445+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad8120ad0 con 0x7f2ab800ffa0 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.445+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2ac0004290 con 0x7f2ab800ffa0 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.445+0000 7f2ad4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.445+0000 7f2ad4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.445+0000 7f2ad4ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:13.447 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:13.448 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.445+0000 7f2ad4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:13.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3505669541 0 0) 0x7f2ad8120ad0 con 0x7f2ab800ffa0 2024-03-25T18:28:13.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 >> v1:172.21.15.55:6790/0 conn(0x7f2ad80a96b0 legacy=0x7f2ad8139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:13.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 >> v1:172.21.15.133:6789/0 conn(0x7f2ab800a700 legacy=0x7f2ad80b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:13.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:13.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 1 monclient: found mon.a 2024-03-25T18:28:13.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:13.457 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad81db960 con 0x7f2ab800ffa0 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2ad4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2add8f6640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2add8f6640 10 monclient: _renew_subs 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2add8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2add8f6640 1 -- 172.21.15.55:0/593489309 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad80b0b10 con 0x7f2ab800ffa0 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2add8f6640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2add8f6640 10 monclient: _renew_subs 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2add8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:13.458 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2add8f6640 1 -- 172.21.15.55:0/593489309 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad813a420 con 0x7f2ab800ffa0 2024-03-25T18:28:13.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.456+0000 7f2add8f6640 1 librados: init done 2024-03-25T18:28:13.459 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.457+0000 7f2add8f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:13.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.457+0000 7f2add8f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:13.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.457+0000 7f2add8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:13.460 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.457+0000 7f2add8f6640 1 -- 172.21.15.55:0/593489309 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad813a420 con 0x7f2ab800ffa0 2024-03-25T18:28:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ac0004710 con 0x7f2ab800ffa0 2024-03-25T18:28:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2ac0004b20 con 0x7f2ab800ffa0 2024-03-25T18:28:13.647 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 10 monclient: dump: 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2ac00050d0 con 0x7f2ab800ffa0 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad5ffb640 4 set_mon_vals no callback set 2024-03-25T18:28:13.648 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2046662693 0 0) 0x7f2ac0006590 con 0x7f2ab800ffa0 2024-03-25T18:28:13.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 10 client.4107.objecter ms_dispatch 0x7f2ad8000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:28:13.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:28:13.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:28:13.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:28:13.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2ac00069b0 con 0x7f2ab800ffa0 2024-03-25T18:28:13.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:13.649 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.646+0000 7f2ad4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:13.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.713+0000 7f2add8f6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:28:13.714 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.713+0000 7f2add8f6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:28:13.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.713+0000 7f2add8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:13.715 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.713+0000 7f2add8f6640 1 -- 172.21.15.55:0/593489309 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2ad813a420 con 0x7f2ab800ffa0 2024-03-25T18:28:13.950 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.949+0000 7f2ad4ff9640 1 -- 172.21.15.55:0/593489309 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f2ac0005690 con 0x7f2ab800ffa0 2024-03-25T18:28:13.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.949+0000 7f2ad4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:28:13.951 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.949+0000 7f2ad4ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:28:13.951 INFO:teuthology.orchestra.run.smithi055.stderr:adjusted tunables profile to default 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 10 librados: watch_flush enter 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 10 librados: watch_flush exit 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 10 monclient: shutdown 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 1 -- 172.21.15.55:0/593489309 >> v1:172.21.15.55:6789/0 conn(0x7f2ab800ffa0 legacy=0x7f2ad8139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 1 -- 172.21.15.55:0/593489309 shutdown_connections 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 1 -- 172.21.15.55:0/593489309 >> 172.21.15.55:0/593489309 conn(0x7f2ad80668e0 msgr2=0x7f2ad80b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 1 -- 172.21.15.55:0/593489309 shutdown_connections 2024-03-25T18:28:13.952 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 1 -- 172.21.15.55:0/593489309 wait complete. 2024-03-25T18:28:13.953 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:13.950+0000 7f2add8f6640 1 librados: shutdown 2024-03-25T18:28:13.962 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:28:13.962 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:28:13.963 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:28:13.963 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:28:13.994 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:28:13.994 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:28:14.060 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:28:14.061 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:28:14.129 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-03-25T18:28:14.129 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:28:14.197 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:28:14.197 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:28:14.224 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:28:14.224 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:28:14.231 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:14.230+0000 7fa1e3641040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:28:14.254 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:14.252+0000 7f6a26412040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:28:14.280 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:28:14.280 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:28:14.296 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:14.295+0000 7fa1e3641040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:28:14.311 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:14.310+0000 7f6a26412040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:28:14.337 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-25T18:28:14.337 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:28:14.403 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new c847a581-38f4-4880-a9a7-ec6492573f17 0 2024-03-25T18:28:14.543 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.541+0000 7f90af37b640 10 monclient: get_monmap_and_config 2024-03-25T18:28:14.543 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.541+0000 7f90af37b640 10 monclient: build_initial_monmap 2024-03-25T18:28:14.543 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.541+0000 7f90af37b640 10 monclient: monmap: 2024-03-25T18:28:14.543 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:14.543 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:14.543 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:28:14.542947+0000 2024-03-25T18:28:14.543 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:28:14.542947+0000 2024-03-25T18:28:14.544 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:14.544 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:14.544 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:14.544 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:14.544 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:14.544 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:14.544 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 1 Processor -- start 2024-03-25T18:28:14.544 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 1 -- start start 2024-03-25T18:28:14.544 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 10 monclient: init 2024-03-25T18:28:14.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 10 monclient(hunting): picked mon.noname-b con 0x7f90a80b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 10 monclient(hunting): picked mon.noname-c con 0x7f90a8142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 10 monclient(hunting): picked mon.noname-a con 0x7f90a80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a80a9d60 con 0x7f90a80a96b0 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a813ff10 con 0x7f90a8142a20 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a80aa1f0 con 0x7f90a80b4700 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.542+0000 7f90af37b640 10 monclient(hunting): authenticate will time out at 7327.705650s 2024-03-25T18:28:14.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.543+0000 7f90ac916640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f90a8142a20 0x7f90a813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58220/0 (socket says 172.21.15.133:58220) 2024-03-25T18:28:14.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.543+0000 7f90ac916640 1 -- 172.21.15.133:0/985802153 learned_addr learned my addr 172.21.15.133:0/985802153 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:14.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.545+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999230552 0 0) 0x7f90a813ff10 con 0x7f90a8142a20 2024-03-25T18:28:14.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.545+0000 7f909ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.545+0000 7f909ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.545+0000 7f909ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.545+0000 7f909ffff640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:28:14.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.545+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9094003520 con 0x7f90a8142a20 2024-03-25T18:28:14.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903415797 0 0) 0x7f90a80aa1f0 con 0x7f90a80b4700 2024-03-25T18:28:14.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:28:14.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a813ff10 con 0x7f90a80b4700 2024-03-25T18:28:14.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760099485 0 0) 0x7f90a80a9d60 con 0x7f90a80a96b0 2024-03-25T18:28:14.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:28:14.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.546+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a80aa1f0 con 0x7f90a80a96b0 2024-03-25T18:28:14.550 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3559384813 0 0) 0x7f9094003520 con 0x7f90a8142a20 2024-03-25T18:28:14.550 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 >> v1:172.21.15.55:6790/0 conn(0x7f90a80b4700 legacy=0x7f90a8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 >> v1:172.21.15.55:6789/0 conn(0x7f90a80a96b0 legacy=0x7f90a813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 1 monclient: found mon.noname-c 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a80b4ae0 con 0x7f90a8142a20 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 10 monclient: _finish_auth 0 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9098001690 con 0x7f90a8142a20 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f909ffff640 10 monclient: dump: 2024-03-25T18:28:14.551 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f90af37b640 5 monclient: authenticate success, global_id 4117 2024-03-25T18:28:14.552 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.549+0000 7f90af37b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:14.576 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.575+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90980017f0 con 0x7f90a8142a20 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.575+0000 7f909ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.575+0000 7f909ffff640 1 -- 172.21.15.133:0/985802153 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9098001c00 con 0x7f90a8142a20 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.575+0000 7f909ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.575+0000 7f909ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.575+0000 7f909ffff640 10 monclient: dump: 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:14.577 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:14.578 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:14.578 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.575+0000 7f90af37b640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:14.578 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.575+0000 7f90af37b640 4 set_mon_vals no callback set 2024-03-25T18:28:14.579 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.576+0000 7f90af37b640 10 monclient: shutdown 2024-03-25T18:28:14.579 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.576+0000 7f90af37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:14.579 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.576+0000 7f90af37b640 1 -- 172.21.15.133:0/985802153 >> v1:172.21.15.133:6789/0 conn(0x7f90a8142a20 legacy=0x7f90a813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.579 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.576+0000 7f90af37b640 1 -- 172.21.15.133:0/985802153 shutdown_connections 2024-03-25T18:28:14.579 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.576+0000 7f90af37b640 1 -- 172.21.15.133:0/985802153 >> 172.21.15.133:0/985802153 conn(0x7f90a80668e0 msgr2=0x7f90a8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.576+0000 7f90af37b640 1 -- 172.21.15.133:0/985802153 shutdown_connections 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.576+0000 7f90af37b640 1 -- 172.21.15.133:0/985802153 wait complete. 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.577+0000 7f90af37b640 10 monclient: build_initial_monmap 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.577+0000 7f90af37b640 10 monclient: monmap: 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:14.580 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.577+0000 7f90af37b640 1 librados: starting msgr at 2024-03-25T18:28:14.581 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.577+0000 7f90af37b640 1 librados: starting objecter 2024-03-25T18:28:14.581 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.577+0000 7f90af37b640 1 Processor -- start 2024-03-25T18:28:14.581 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 1 -- start start 2024-03-25T18:28:14.581 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 1 librados: setting wanted keys 2024-03-25T18:28:14.581 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 1 librados: calling monclient init 2024-03-25T18:28:14.581 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 10 monclient: init 2024-03-25T18:28:14.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:14.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:14.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 10 monclient(hunting): picked mon.noname-a con 0x7f90a80a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:14.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 10 monclient(hunting): picked mon.noname-b con 0x7f90a80b4700 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:14.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 10 monclient(hunting): picked mon.noname-c con 0x7f90a8142a20 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:14.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a80ade40 con 0x7f90a80a96b0 2024-03-25T18:28:14.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.578+0000 7f90af37b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a80ae0b0 con 0x7f90a80b4700 2024-03-25T18:28:14.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.579+0000 7f90af37b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a80ae390 con 0x7f90a8142a20 2024-03-25T18:28:14.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.579+0000 7f90af37b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.579+0000 7f90af37b640 10 monclient(hunting): authenticate will time out at 7327.741736s 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.579+0000 7f90ac916640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f90a80b4700 0x7f90a80ae7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58224/0 (socket says 172.21.15.133:58224) 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.579+0000 7f90ac916640 1 -- 172.21.15.133:0/1056938069 learned_addr learned my addr 172.21.15.133:0/1056938069 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.579+0000 7f909dffb640 10 client.?.objecter ms_handle_connect 0x7f90a80b4700 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.579+0000 7f909dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.580+0000 7f909dffb640 10 client.?.objecter ms_handle_connect 0x7f90a80a96b0 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.580+0000 7f909dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.580+0000 7f909dffb640 10 client.?.objecter ms_handle_connect 0x7f90a8142a20 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.580+0000 7f909dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.581+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384923222 0 0) 0x7f90a80ae0b0 con 0x7f90a80b4700 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.581+0000 7f909dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.581+0000 7f909dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.581+0000 7f909dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.581+0000 7f909dffb640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:28:14.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.581+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c004eb0 con 0x7f90a80b4700 2024-03-25T18:28:14.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.582+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855002469 0 0) 0x7f90a80ae390 con 0x7f90a8142a20 2024-03-25T18:28:14.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.582+0000 7f909dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.582+0000 7f909dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.582+0000 7f909dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.582+0000 7f909dffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:28:14.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.582+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a80ae0b0 con 0x7f90a8142a20 2024-03-25T18:28:14.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.583+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942007439 0 0) 0x7f90a80ade40 con 0x7f90a80a96b0 2024-03-25T18:28:14.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.583+0000 7f909dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:14.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.583+0000 7f909dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:14.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.583+0000 7f909dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:14.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.583+0000 7f909dffb640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:28:14.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.583+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a80ae390 con 0x7f90a80a96b0 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.586+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191737674 0 0) 0x7f908c004eb0 con 0x7f90a80b4700 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.586+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90a80ade40 con 0x7f90a80b4700 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.586+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9098001bc0 con 0x7f90a80b4700 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.586+0000 7f909dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.586+0000 7f909dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.586+0000 7f909dffb640 10 monclient(hunting): dump: 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:14.588 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:14.589 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:14.589 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:14.589 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:14.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.586+0000 7f909dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:14.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.590+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842799932 0 0) 0x7f90a80ae390 con 0x7f90a80a96b0 2024-03-25T18:28:14.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.590+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f908c004eb0 con 0x7f90a80a96b0 2024-03-25T18:28:14.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.590+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f90a40018f0 con 0x7f90a80a96b0 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.590+0000 7f909dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.590+0000 7f909dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.590+0000 7f909dffb640 10 monclient(hunting): dump: 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:14.592 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:14.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4142446864 0 0) 0x7f908c004eb0 con 0x7f90a80a96b0 2024-03-25T18:28:14.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 >> v1:172.21.15.55:6790/0 conn(0x7f90a8142a20 legacy=0x7f90a81e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 >> v1:172.21.15.133:6789/0 conn(0x7f90a80b4700 legacy=0x7f90a80ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:14.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 1 monclient: found mon.a 2024-03-25T18:28:14.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:14.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:14.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a80ad060 con 0x7f90a80a96b0 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f909dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f90af37b640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f90af37b640 10 monclient: _renew_subs 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f90af37b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f90af37b640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a80abcc0 con 0x7f90a80a96b0 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f90af37b640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f90af37b640 10 monclient: _renew_subs 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f90af37b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f90af37b640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a80acb40 con 0x7f90a80a96b0 2024-03-25T18:28:14.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.594+0000 7f90af37b640 1 librados: init done 2024-03-25T18:28:14.598 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.595+0000 7f90af37b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:14.598 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.595+0000 7f90af37b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:14.598 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.595+0000 7f90af37b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:14.598 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.595+0000 7f90af37b640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a80ad2a0 con 0x7f90a80a96b0 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.747+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90a40035f0 con 0x7f90a80a96b0 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.747+0000 7f909dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.747+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f90a4003760 con 0x7f90a80a96b0 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.747+0000 7f909dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.747+0000 7f909dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.747+0000 7f909dffb640 10 monclient: dump: 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.749 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:14.750 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:14.750 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:14.750 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:14.750 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:14.750 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:14.750 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:14.750 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.747+0000 7f909f7fe640 4 set_mon_vals no callback set 2024-03-25T18:28:14.750 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.747+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f90a4003c50 con 0x7f90a80a96b0 2024-03-25T18:28:14.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.748+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 408543155 0 0) 0x7f90a40050b0 con 0x7f90a80a96b0 2024-03-25T18:28:14.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.748+0000 7f909dffb640 10 client.4119.objecter ms_dispatch 0x7f90a8000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:28:14.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.748+0000 7f909dffb640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:28:14.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.748+0000 7f909dffb640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:28:14.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.748+0000 7f909dffb640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:28:14.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.748+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f90a4004100 con 0x7f90a80a96b0 2024-03-25T18:28:14.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.748+0000 7f909dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:14.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.748+0000 7f909dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:14.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.820+0000 7f90af37b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c847a581-38f4-4880-a9a7-ec6492573f17", "id": 0}] 2024-03-25T18:28:14.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.820+0000 7f90af37b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c847a581-38f4-4880-a9a7-ec6492573f17", "id": 0}] 2024-03-25T18:28:14.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.820+0000 7f90af37b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:14.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.820+0000 7f90af37b640 1 -- 172.21.15.133:0/1056938069 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c847a581-38f4-4880-a9a7-ec6492573f17", "id": 0} v 0) v1 -- 0x7f90a80abfc0 con 0x7f90a80a96b0 2024-03-25T18:28:14.862 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:14.860+0000 7f6a26412040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:28:14.908 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.906+0000 7f909dffb640 1 -- 172.21.15.133:0/1056938069 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c847a581-38f4-4880-a9a7-ec6492573f17", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 234179419 0 732707515) 0x7f90a4004470 con 0x7f90a80a96b0 2024-03-25T18:28:14.908 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.906+0000 7f909dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c847a581-38f4-4880-a9a7-ec6492573f17", "id": 0}] 2024-03-25T18:28:14.908 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.906+0000 7f909dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:14.908 INFO:teuthology.orchestra.run.smithi133.stdout:0 2024-03-25T18:28:14.909 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.907+0000 7f90af37b640 10 librados: watch_flush enter 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.907+0000 7f90af37b640 10 librados: watch_flush exit 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.907+0000 7f90af37b640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.907+0000 7f90af37b640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.907+0000 7f90af37b640 10 monclient: shutdown 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.907+0000 7f90af37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.907+0000 7f90af37b640 1 -- 172.21.15.133:0/1056938069 >> v1:172.21.15.55:6789/0 conn(0x7f90a80a96b0 legacy=0x7f90a813aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.907+0000 7f90af37b640 1 -- 172.21.15.133:0/1056938069 shutdown_connections 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.907+0000 7f90af37b640 1 -- 172.21.15.133:0/1056938069 >> 172.21.15.133:0/1056938069 conn(0x7f90a80668e0 msgr2=0x7f90a80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.908+0000 7f90af37b640 1 -- 172.21.15.133:0/1056938069 shutdown_connections 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.908+0000 7f90af37b640 1 -- 172.21.15.133:0/1056938069 wait complete. 2024-03-25T18:28:14.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:14.908+0000 7f90af37b640 1 librados: shutdown 2024-03-25T18:28:14.915 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:14.915+0000 7fa1e3641040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:28:14.917 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:14.916+0000 7f6a26412040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:28:14.919 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new b0db4fe3-7bc7-4836-8bf4-c9744981c68c 1 2024-03-25T18:28:14.965 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:14.964+0000 7f6a26412040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:28:14.973 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:14.972+0000 7fa1e3641040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:28:15.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.018+0000 7f13e4233640 10 monclient: get_monmap_and_config 2024-03-25T18:28:15.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.018+0000 7f13e4233640 10 monclient: build_initial_monmap 2024-03-25T18:28:15.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.018+0000 7f13e4233640 10 monclient: monmap: 2024-03-25T18:28:15.020 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:15.020 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.020 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:28:15.019505+0000 2024-03-25T18:28:15.020 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:28:15.019505+0000 2024-03-25T18:28:15.020 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:15.020 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.021 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:15.021 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:15.021 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:15.021 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.018+0000 7f13e4233640 1 Processor -- start 2024-03-25T18:28:15.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.018+0000 7f13e4233640 1 -- start start 2024-03-25T18:28:15.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.018+0000 7f13e4233640 10 monclient: init 2024-03-25T18:28:15.021 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:15.020+0000 7fa1e3641040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:28:15.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:15.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 10 monclient(hunting): picked mon.noname-c con 0x7f13dc0b4700 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 10 monclient(hunting): picked mon.noname-a con 0x7f13dc142a20 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 10 monclient(hunting): picked mon.noname-b con 0x7f13dc0a96b0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc0a9d60 con 0x7f13dc142a20 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc13ff10 con 0x7f13dc0b4700 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc0aa1f0 con 0x7f13dc0a96b0 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e4233640 10 monclient(hunting): authenticate will time out at 7328.182203s 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e1fcf640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f13dc0b4700 0x7f13dc142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58228/0 (socket says 172.21.15.133:58228) 2024-03-25T18:28:15.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.019+0000 7f13e1fcf640 1 -- 172.21.15.133:0/122499830 learned_addr learned my addr 172.21.15.133:0/122499830 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:15.024 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:15.021+0000 7f6a26412040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213589389 0 0) 0x7f13dc13ff10 con 0x7f13dc0b4700 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c8003520 con 0x7f13dc0b4700 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101185497 0 0) 0x7f13dc0a9d60 con 0x7f13dc142a20 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13dc13ff10 con 0x7f13dc142a20 2024-03-25T18:28:15.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893704197 0 0) 0x7f13dc0aa1f0 con 0x7f13dc0a96b0 2024-03-25T18:28:15.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:28:15.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.022+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13dc0a9d60 con 0x7f13dc0a96b0 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2400601136 0 0) 0x7f13c8003520 con 0x7f13dc0b4700 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 >> v1:172.21.15.55:6790/0 conn(0x7f13dc0a96b0 legacy=0x7f13dc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 >> v1:172.21.15.55:6789/0 conn(0x7f13dc142a20 legacy=0x7f13dc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 1 monclient: found mon.noname-c 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc0b4ae0 con 0x7f13dc0b4700 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 10 monclient: _finish_auth 0 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:15.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f13cc001690 con 0x7f13dc0b4700 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e0fcd640 10 monclient: dump: 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e4233640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:28:15.029 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.025+0000 7f13e4233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:15.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.027+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13cc0017f0 con 0x7f13dc0b4700 2024-03-25T18:28:15.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.027+0000 7f13e0fcd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:15.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.027+0000 7f13e0fcd640 1 -- 172.21.15.133:0/122499830 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f13cc001c00 con 0x7f13dc0b4700 2024-03-25T18:28:15.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.027+0000 7f13e0fcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.032 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.027+0000 7f13e0fcd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.027+0000 7f13e0fcd640 10 monclient: dump: 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.033 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.027+0000 7f13e4233640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.027+0000 7f13e4233640 4 set_mon_vals no callback set 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.028+0000 7f13e4233640 10 monclient: shutdown 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.028+0000 7f13e4233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.028+0000 7f13e4233640 1 -- 172.21.15.133:0/122499830 >> v1:172.21.15.133:6789/0 conn(0x7f13dc0b4700 legacy=0x7f13dc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.028+0000 7f13e4233640 1 -- 172.21.15.133:0/122499830 shutdown_connections 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.028+0000 7f13e4233640 1 -- 172.21.15.133:0/122499830 >> 172.21.15.133:0/122499830 conn(0x7f13dc0668e0 msgr2=0x7f13dc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.028+0000 7f13e4233640 1 -- 172.21.15.133:0/122499830 shutdown_connections 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.028+0000 7f13e4233640 1 -- 172.21.15.133:0/122499830 wait complete. 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.029+0000 7f13e4233640 10 monclient: build_initial_monmap 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.029+0000 7f13e4233640 10 monclient: monmap: 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:15.034 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.029+0000 7f13e4233640 1 librados: starting msgr at 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.029+0000 7f13e4233640 1 librados: starting objecter 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.029+0000 7f13e4233640 1 Processor -- start 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.029+0000 7f13e4233640 1 -- start start 2024-03-25T18:28:15.035 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.029+0000 7f13e4233640 1 librados: setting wanted keys 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.029+0000 7f13e4233640 1 librados: calling monclient init 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.029+0000 7f13e4233640 10 monclient: init 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 10 monclient(hunting): picked mon.noname-b con 0x7f13dc0a96b0 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 10 monclient(hunting): picked mon.noname-c con 0x7f13dc0b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 10 monclient(hunting): picked mon.noname-a con 0x7f13dc142a20 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc139660 con 0x7f13dc142a20 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc13a6e0 con 0x7f13dc0a96b0 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13dc1398d0 con 0x7f13dc0b4700 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e4233640 10 monclient(hunting): authenticate will time out at 7328.192919s 2024-03-25T18:28:15.036 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e1fcf640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f13dc0a96b0 0x7f13dc138030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58236/0 (socket says 172.21.15.133:58236) 2024-03-25T18:28:15.037 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13e1fcf640 1 -- 172.21.15.133:0/2013292311 learned_addr learned my addr 172.21.15.133:0/2013292311 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:15.037 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13d27fc640 10 client.?.objecter ms_handle_connect 0x7f13dc0a96b0 2024-03-25T18:28:15.037 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.030+0000 7f13d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.037 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.031+0000 7f13d27fc640 10 client.?.objecter ms_handle_connect 0x7f13dc142a20 2024-03-25T18:28:15.037 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.031+0000 7f13d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.037 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.031+0000 7f13d27fc640 10 client.?.objecter ms_handle_connect 0x7f13dc0b4700 2024-03-25T18:28:15.037 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.031+0000 7f13d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.032+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976424122 0 0) 0x7f13dc13a6e0 con 0x7f13dc0a96b0 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.032+0000 7f13d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.032+0000 7f13d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.032+0000 7f13d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.032+0000 7f13d27fc640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.032+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0004eb0 con 0x7f13dc0a96b0 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835151983 0 0) 0x7f13dc1398d0 con 0x7f13dc0b4700 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13dc13a6e0 con 0x7f13dc0b4700 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301573975 0 0) 0x7f13dc139660 con 0x7f13dc142a20 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.033+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13dc1398d0 con 0x7f13dc142a20 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.037+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697733141 0 0) 0x7f13c0004eb0 con 0x7f13dc0a96b0 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.037+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13dc139660 con 0x7f13dc0a96b0 2024-03-25T18:28:15.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.037+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f13cc001bc0 con 0x7f13dc0a96b0 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.037+0000 7f13d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.037+0000 7f13d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.037+0000 7f13d27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.042 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.037+0000 7f13d27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.037+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1580745303 0 0) 0x7f13dc13a6e0 con 0x7f13dc0b4700 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.037+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13c0004eb0 con 0x7f13dc0b4700 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1144041417 0 0) 0x7f13dc1398d0 con 0x7f13dc142a20 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13dc13a6e0 con 0x7f13dc142a20 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f13d4001bd0 con 0x7f13dc142a20 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.043 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f13d8001990 con 0x7f13dc0b4700 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.038+0000 7f13d27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.044 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.045 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.045 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.045 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.045 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.045 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.045 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.045 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.045 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.046 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2068445389 0 0) 0x7f13dc13a6e0 con 0x7f13dc142a20 2024-03-25T18:28:15.046 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 >> v1:172.21.15.55:6790/0 conn(0x7f13dc0b4700 legacy=0x7f13dc139fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 >> v1:172.21.15.133:6789/0 conn(0x7f13dc0a96b0 legacy=0x7f13dc138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 1 monclient: found mon.a 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc139bb0 con 0x7f13dc142a20 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13e4233640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13e4233640 10 monclient: _renew_subs 2024-03-25T18:28:15.047 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13e4233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13e4233640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc138900 con 0x7f13dc142a20 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13e4233640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13e4233640 10 monclient: _renew_subs 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13e4233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13e4233640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc135ab0 con 0x7f13dc142a20 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.040+0000 7f13e4233640 1 librados: init done 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.041+0000 7f13e4233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.041+0000 7f13e4233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.041+0000 7f13e4233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.048 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.041+0000 7f13e4233640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc04f730 con 0x7f13dc142a20 2024-03-25T18:28:15.068 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:15.067+0000 7f6a26412040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:28:15.078 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:15.078+0000 7fa1e3641040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:28:15.123 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:15.122+0000 7fa1e3641040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:28:15.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.149+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d4001fe0 con 0x7f13dc142a20 2024-03-25T18:28:15.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.149+0000 7f13d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:15.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.149+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f13d4003b30 con 0x7f13dc142a20 2024-03-25T18:28:15.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.149+0000 7f13d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.149+0000 7f13d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.149+0000 7f13d27fc640 10 monclient: dump: 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.151 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.149+0000 7f13d3fff640 4 set_mon_vals no callback set 2024-03-25T18:28:15.152 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.149+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f13d4004020 con 0x7f13dc142a20 2024-03-25T18:28:15.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.150+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2131906549 0 0) 0x7f13d40054a0 con 0x7f13dc142a20 2024-03-25T18:28:15.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.150+0000 7f13d27fc640 10 client.4125.objecter ms_dispatch 0x7f13dc000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:28:15.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.150+0000 7f13d27fc640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:28:15.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.150+0000 7f13d27fc640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:28:15.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.150+0000 7f13d27fc640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:28:15.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.150+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f13d4005810 con 0x7f13dc142a20 2024-03-25T18:28:15.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.150+0000 7f13d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.150+0000 7f13d27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:15.187 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:15.185+0000 7f6a26412040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:28:15.224 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.222+0000 7f13e4233640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b0db4fe3-7bc7-4836-8bf4-c9744981c68c", "id": 1}] 2024-03-25T18:28:15.224 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.222+0000 7f13e4233640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b0db4fe3-7bc7-4836-8bf4-c9744981c68c", "id": 1}] 2024-03-25T18:28:15.224 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.222+0000 7f13e4233640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.224 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.222+0000 7f13e4233640 1 -- 172.21.15.133:0/2013292311 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b0db4fe3-7bc7-4836-8bf4-c9744981c68c", "id": 1} v 0) v1 -- 0x7f13dc13e650 con 0x7f13dc142a20 2024-03-25T18:28:15.244 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:15.243+0000 7fa1e3641040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:28:15.259 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:15.258+0000 7f6a26412040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:28:15.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.260+0000 7f13d27fc640 1 -- 172.21.15.133:0/2013292311 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b0db4fe3-7bc7-4836-8bf4-c9744981c68c", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3424328192 0 940483276) 0x7f13d4005a50 con 0x7f13dc142a20 2024-03-25T18:28:15.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.260+0000 7f13d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b0db4fe3-7bc7-4836-8bf4-c9744981c68c", "id": 1}] 2024-03-25T18:28:15.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.260+0000 7f13d27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:15.262 INFO:teuthology.orchestra.run.smithi133.stdout:1 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 10 librados: watch_flush enter 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 10 librados: watch_flush exit 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 10 monclient: shutdown 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 1 -- 172.21.15.133:0/2013292311 >> v1:172.21.15.55:6789/0 conn(0x7f13dc142a20 legacy=0x7f13dc1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 1 -- 172.21.15.133:0/2013292311 shutdown_connections 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 1 -- 172.21.15.133:0/2013292311 >> 172.21.15.133:0/2013292311 conn(0x7f13dc0668e0 msgr2=0x7f13dc0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 1 -- 172.21.15.133:0/2013292311 shutdown_connections 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 1 -- 172.21.15.133:0/2013292311 wait complete. 2024-03-25T18:28:15.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.261+0000 7f13e4233640 1 librados: shutdown 2024-03-25T18:28:15.273 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new 2effb575-2dc6-44ea-bc99-c85b6c87643c 2 2024-03-25T18:28:15.310 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:15.309+0000 7f6a26412040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:28:15.320 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:15.319+0000 7fa1e3641040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:28:15.366 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:15.364+0000 7f6a26412040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:28:15.367 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:15.367+0000 7fa1e3641040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:28:15.372 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.371+0000 7fe4267f6640 10 monclient: get_monmap_and_config 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.371+0000 7fe4267f6640 10 monclient: build_initial_monmap 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.371+0000 7fe4267f6640 10 monclient: monmap: 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:28:15.373041+0000 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:28:15.373041+0000 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:15.373 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 1 Processor -- start 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 1 -- start start 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 10 monclient: init 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe4200b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe420142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe4200a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4200a9d60 con 0x7fe4200a96b0 2024-03-25T18:28:15.375 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42013ff10 con 0x7fe420142a20 2024-03-25T18:28:15.376 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4200aa1f0 con 0x7fe4200b4700 2024-03-25T18:28:15.376 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:15.376 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.372+0000 7fe4267f6640 10 monclient(hunting): authenticate will time out at 7328.535690s 2024-03-25T18:28:15.376 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.373+0000 7fe41f7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fe420142a20 0x7fe42013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58238/0 (socket says 172.21.15.133:58238) 2024-03-25T18:28:15.376 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.373+0000 7fe41f7fe640 1 -- 172.21.15.133:0/3878688960 learned_addr learned my addr 172.21.15.133:0/3878688960 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:15.377 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042985881 0 0) 0x7fe42013ff10 con 0x7fe420142a20 2024-03-25T18:28:15.377 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.377 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.377 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.377 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:28:15.378 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe408003520 con 0x7fe420142a20 2024-03-25T18:28:15.378 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087115553 0 0) 0x7fe4200aa1f0 con 0x7fe4200b4700 2024-03-25T18:28:15.378 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.378 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.378 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.378 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:28:15.379 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.376+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42013ff10 con 0x7fe4200b4700 2024-03-25T18:28:15.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926522115 0 0) 0x7fe408003520 con 0x7fe420142a20 2024-03-25T18:28:15.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 >> v1:172.21.15.55:6790/0 conn(0x7fe4200b4700 legacy=0x7fe420142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 >> v1:172.21.15.55:6789/0 conn(0x7fe4200a96b0 legacy=0x7fe42013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:15.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 1 monclient: found mon.noname-c 2024-03-25T18:28:15.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:15.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:28:15.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4200b4ae0 con 0x7fe420142a20 2024-03-25T18:28:15.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 10 monclient: _finish_auth 0 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fe410001690 con 0x7fe420142a20 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe41effd640 10 monclient: dump: 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.382 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.383 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.383 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe4267f6640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:28:15.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.379+0000 7fe4267f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.381+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4100017f0 con 0x7fe420142a20 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.381+0000 7fe41effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.381+0000 7fe41effd640 1 -- 172.21.15.133:0/3878688960 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fe410001c00 con 0x7fe420142a20 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.381+0000 7fe4267f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.381+0000 7fe4267f6640 4 set_mon_vals no callback set 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.381+0000 7fe41effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.381+0000 7fe41effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.381+0000 7fe41effd640 10 monclient: dump: 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.385 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.382+0000 7fe4267f6640 10 monclient: shutdown 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.382+0000 7fe4267f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.382+0000 7fe4267f6640 1 -- 172.21.15.133:0/3878688960 >> v1:172.21.15.133:6789/0 conn(0x7fe420142a20 legacy=0x7fe42013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.382+0000 7fe4267f6640 1 -- 172.21.15.133:0/3878688960 shutdown_connections 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.382+0000 7fe4267f6640 1 -- 172.21.15.133:0/3878688960 >> 172.21.15.133:0/3878688960 conn(0x7fe4200668e0 msgr2=0x7fe420066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.382+0000 7fe4267f6640 1 -- 172.21.15.133:0/3878688960 shutdown_connections 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.382+0000 7fe4267f6640 1 -- 172.21.15.133:0/3878688960 wait complete. 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.383+0000 7fe4267f6640 10 monclient: build_initial_monmap 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.383+0000 7fe4267f6640 10 monclient: monmap: 2024-03-25T18:28:15.386 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.383+0000 7fe4267f6640 1 librados: starting msgr at 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.383+0000 7fe4267f6640 1 librados: starting objecter 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.383+0000 7fe4267f6640 1 Processor -- start 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.383+0000 7fe4267f6640 1 -- start start 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.383+0000 7fe4267f6640 1 librados: setting wanted keys 2024-03-25T18:28:15.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.383+0000 7fe4267f6640 1 librados: calling monclient init 2024-03-25T18:28:15.388 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.383+0000 7fe4267f6640 10 monclient: init 2024-03-25T18:28:15.389 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe4200a96b0 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe4200b4700 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe420142a20 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4200ac210 con 0x7fe4200b4700 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42013c910 con 0x7fe4200a96b0 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4200acf90 con 0x7fe420142a20 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe4267f6640 10 monclient(hunting): authenticate will time out at 7328.547015s 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe41ffff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fe4200a96b0 0x7fe420139dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58246/0 (socket says 172.21.15.133:58246) 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.384+0000 7fe41ffff640 1 -- 172.21.15.133:0/2203062875 learned_addr learned my addr 172.21.15.133:0/2203062875 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.385+0000 7fe41cff9640 10 client.?.objecter ms_handle_connect 0x7fe4200a96b0 2024-03-25T18:28:15.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.385+0000 7fe41cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.385+0000 7fe41cff9640 10 client.?.objecter ms_handle_connect 0x7fe420142a20 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.385+0000 7fe41cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.387+0000 7fe41cff9640 10 client.?.objecter ms_handle_connect 0x7fe4200b4700 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.387+0000 7fe41cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.387+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127055944 0 0) 0x7fe4200acf90 con 0x7fe420142a20 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.387+0000 7fe41cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.387+0000 7fe41cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.387+0000 7fe41cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.387+0000 7fe41cff9640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.387+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe40c004eb0 con 0x7fe420142a20 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.388+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289928465 0 0) 0x7fe42013c910 con 0x7fe4200a96b0 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.388+0000 7fe41cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.388+0000 7fe41cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.391 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.388+0000 7fe41cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.392 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.388+0000 7fe41cff9640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:28:15.392 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.388+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4200acf90 con 0x7fe4200a96b0 2024-03-25T18:28:15.392 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.389+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909402769 0 0) 0x7fe4200ac210 con 0x7fe4200b4700 2024-03-25T18:28:15.392 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.389+0000 7fe41cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.392 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.389+0000 7fe41cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.392 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.389+0000 7fe41cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.392 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.389+0000 7fe41cff9640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:28:15.393 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.389+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42013c910 con 0x7fe4200b4700 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.394+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849965991 0 0) 0x7fe42013c910 con 0x7fe4200b4700 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.394+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4200ac210 con 0x7fe4200b4700 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.394+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fe410001b00 con 0x7fe4200b4700 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.394+0000 7fe41cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.394+0000 7fe41cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.394+0000 7fe41cff9640 10 monclient(hunting): dump: 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.396 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.397 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.397 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.397 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.394+0000 7fe41cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:15.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 100970956 0 0) 0x7fe4200ac210 con 0x7fe4200b4700 2024-03-25T18:28:15.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 >> v1:172.21.15.55:6790/0 conn(0x7fe420142a20 legacy=0x7fe4200ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 >> v1:172.21.15.133:6789/0 conn(0x7fe4200a96b0 legacy=0x7fe420139dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:15.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 1 monclient: found mon.a 2024-03-25T18:28:15.402 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4200ac460 con 0x7fe4200b4700 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe41cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe4267f6640 5 monclient: authenticate success, global_id 4128 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe4267f6640 10 monclient: _renew_subs 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe4267f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe4267f6640 1 -- 172.21.15.133:0/2203062875 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42013cc20 con 0x7fe4200b4700 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe4267f6640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe4267f6640 10 monclient: _renew_subs 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe4267f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe4267f6640 1 -- 172.21.15.133:0/2203062875 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42013b630 con 0x7fe4200b4700 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.400+0000 7fe4267f6640 1 librados: init done 2024-03-25T18:28:15.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.401+0000 7fe4267f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.401+0000 7fe4267f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.401+0000 7fe4267f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.401+0000 7fe4267f6640 1 -- 172.21.15.133:0/2203062875 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4200b6c90 con 0x7fe4200b4700 2024-03-25T18:28:15.419 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:15.418+0000 7f6a26412040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:28:15.429 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:15.428+0000 7fa1e3641040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:28:15.442 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.439+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe410001d20 con 0x7fe4200b4700 2024-03-25T18:28:15.442 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.439+0000 7fe41cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:15.442 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.439+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fe410004c30 con 0x7fe4200b4700 2024-03-25T18:28:15.442 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.439+0000 7fe41cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.439+0000 7fe41cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.439+0000 7fe41cff9640 10 monclient: dump: 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.439+0000 7fe41e7fc640 4 set_mon_vals no callback set 2024-03-25T18:28:15.478 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:15.477+0000 7fa1e3641040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:28:15.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.545+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe410005120 con 0x7fe4200b4700 2024-03-25T18:28:15.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.545+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2623749037 0 0) 0x7fe4100066d0 con 0x7fe4200b4700 2024-03-25T18:28:15.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.545+0000 7fe41cff9640 10 client.4128.objecter ms_dispatch 0x7fe420000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:28:15.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.545+0000 7fe41cff9640 3 client.4128.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:28:15.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.545+0000 7fe41cff9640 3 client.4128.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:28:15.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.545+0000 7fe41cff9640 20 client.4128.objecter dump_active .. 0 homeless 2024-03-25T18:28:15.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.546+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe410006360 con 0x7fe4200b4700 2024-03-25T18:28:15.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.546+0000 7fe41cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.546+0000 7fe41cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:15.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.620+0000 7fe4267f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2effb575-2dc6-44ea-bc99-c85b6c87643c", "id": 2}] 2024-03-25T18:28:15.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.620+0000 7fe4267f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2effb575-2dc6-44ea-bc99-c85b6c87643c", "id": 2}] 2024-03-25T18:28:15.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.620+0000 7fe4267f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:15.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.620+0000 7fe4267f6640 1 -- 172.21.15.133:0/2203062875 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2effb575-2dc6-44ea-bc99-c85b6c87643c", "id": 2} v 0) v1 -- 0x7fe42004f730 con 0x7fe4200b4700 2024-03-25T18:28:15.669 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.668+0000 7fe41cff9640 1 -- 172.21.15.133:0/2203062875 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2effb575-2dc6-44ea-bc99-c85b6c87643c", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1131004552 0 216599125) 0x7fe4100068e0 con 0x7fe4200b4700 2024-03-25T18:28:15.669 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.668+0000 7fe41cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2effb575-2dc6-44ea-bc99-c85b6c87643c", "id": 2}] 2024-03-25T18:28:15.669 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.668+0000 7fe41cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:15.669 INFO:teuthology.orchestra.run.smithi133.stdout:2 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 10 librados: watch_flush enter 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 10 librados: watch_flush exit 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 20 client.4128.objecter shutdown clearing up homeless session... 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 10 client.4128.objecter successfully canceled tick 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 10 monclient: shutdown 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 1 -- 172.21.15.133:0/2203062875 >> v1:172.21.15.55:6789/0 conn(0x7fe4200b4700 legacy=0x7fe42013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 1 -- 172.21.15.133:0/2203062875 shutdown_connections 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 1 -- 172.21.15.133:0/2203062875 >> 172.21.15.133:0/2203062875 conn(0x7fe4200668e0 msgr2=0x7fe4200b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:15.671 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 1 -- 172.21.15.133:0/2203062875 shutdown_connections 2024-03-25T18:28:15.672 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 1 -- 172.21.15.133:0/2203062875 wait complete. 2024-03-25T18:28:15.672 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.669+0000 7fe4267f6640 1 librados: shutdown 2024-03-25T18:28:15.684 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new 6d01598c-3cbe-4167-a7e5-606491d1eac8 3 2024-03-25T18:28:15.820 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:15.818+0000 7f6a26412040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:28:15.824 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.823+0000 7fd9d9196640 10 monclient: get_monmap_and_config 2024-03-25T18:28:15.824 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.823+0000 7fd9d9196640 10 monclient: build_initial_monmap 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.823+0000 7fd9d9196640 10 monclient: monmap: 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:28:15.824917+0000 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:28:15.824917+0000 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:15.825 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 1 Processor -- start 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 1 -- start start 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 10 monclient: init 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d4070210 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d40b06d0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d40705f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40b0b80 con 0x7fd9d40b06d0 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40b0430 con 0x7fd9d4070210 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4071670 con 0x7fd9d40705f0 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:15.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.824+0000 7fd9d9196640 10 monclient(hunting): authenticate will time out at 7328.987378s 2024-03-25T18:28:15.828 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.825+0000 7fd9d2d9d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fd9d4070210 0x7fd9d4140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58262/0 (socket says 172.21.15.133:58262) 2024-03-25T18:28:15.828 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.825+0000 7fd9d2d9d640 1 -- 172.21.15.133:0/2506256538 learned_addr learned my addr 172.21.15.133:0/2506256538 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:15.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148703210 0 0) 0x7fd9c4001110 con 0x7fd9d4070210 2024-03-25T18:28:15.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:28:15.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9bc003520 con 0x7fd9d4070210 2024-03-25T18:28:15.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396152714 0 0) 0x7fd9d40b0b80 con 0x7fd9d40b06d0 2024-03-25T18:28:15.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.829 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:28:15.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c4001110 con 0x7fd9d40b06d0 2024-03-25T18:28:15.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96348837 0 0) 0x7fd9d4071670 con 0x7fd9d40705f0 2024-03-25T18:28:15.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:28:15.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.827+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d40b0b80 con 0x7fd9d40705f0 2024-03-25T18:28:15.832 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314162110 0 0) 0x7fd9bc003520 con 0x7fd9d4070210 2024-03-25T18:28:15.832 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 >> v1:172.21.15.55:6790/0 conn(0x7fd9d40705f0 legacy=0x7fd9d414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.832 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 >> v1:172.21.15.55:6789/0 conn(0x7fd9d40b06d0 legacy=0x7fd9d40afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.832 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:28:15.832 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:28:15.832 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:15.832 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:28:15.832 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d4070a60 con 0x7fd9d4070210 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd9c4001680 con 0x7fd9d4070210 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d1d9b640 10 monclient: dump: 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.833 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.834 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.834 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.834 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d9196640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:28:15.834 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.830+0000 7fd9d9196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:15.836 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9c40017e0 con 0x7fd9d4070210 2024-03-25T18:28:15.836 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:15.836 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d9196640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:15.836 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d9196640 4 set_mon_vals no callback set 2024-03-25T18:28:15.836 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d1d9b640 1 -- 172.21.15.133:0/2506256538 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd9c4001b70 con 0x7fd9d4070210 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d1d9b640 10 monclient: dump: 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d9196640 10 monclient: shutdown 2024-03-25T18:28:15.837 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d9196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d9196640 1 -- 172.21.15.133:0/2506256538 >> v1:172.21.15.133:6789/0 conn(0x7fd9d4070210 legacy=0x7fd9d4140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d9196640 1 -- 172.21.15.133:0/2506256538 shutdown_connections 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d9196640 1 -- 172.21.15.133:0/2506256538 >> 172.21.15.133:0/2506256538 conn(0x7fd9d40b5cd0 msgr2=0x7fd9d40b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d9196640 1 -- 172.21.15.133:0/2506256538 shutdown_connections 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.832+0000 7fd9d9196640 1 -- 172.21.15.133:0/2506256538 wait complete. 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.833+0000 7fd9d9196640 10 monclient: build_initial_monmap 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.833+0000 7fd9d9196640 10 monclient: monmap: 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2024-03-25T18:28:15.838 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.833+0000 7fd9d9196640 1 librados: starting msgr at 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.833+0000 7fd9d9196640 1 librados: starting objecter 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 1 Processor -- start 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 1 -- start start 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 1 librados: setting wanted keys 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 1 librados: calling monclient init 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 10 monclient: init 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d4070210 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d40705f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:15.839 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d40b06d0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:15.840 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4134ab0 con 0x7fd9d40705f0 2024-03-25T18:28:15.840 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d413ad20 con 0x7fd9d4070210 2024-03-25T18:28:15.840 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4143d20 con 0x7fd9d40b06d0 2024-03-25T18:28:15.840 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:15.840 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.834+0000 7fd9d9196640 10 monclient(hunting): authenticate will time out at 7328.997543s 2024-03-25T18:28:15.840 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.835+0000 7fd9d2d9d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fd9d4070210 0x7fd9d41d76c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58264/0 (socket says 172.21.15.133:58264) 2024-03-25T18:28:15.840 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.835+0000 7fd9d2d9d640 1 -- 172.21.15.133:0/784404059 learned_addr learned my addr 172.21.15.133:0/784404059 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:15.840 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.835+0000 7fd9bb7fe640 10 client.?.objecter ms_handle_connect 0x7fd9d4070210 2024-03-25T18:28:15.840 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.835+0000 7fd9bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.841 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.835+0000 7fd9bb7fe640 10 client.?.objecter ms_handle_connect 0x7fd9d40b06d0 2024-03-25T18:28:15.841 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.835+0000 7fd9bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725527856 0 0) 0x7fd9d413ad20 con 0x7fd9d4070210 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c0004eb0 con 0x7fd9d4070210 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890622071 0 0) 0x7fd9d4143d20 con 0x7fd9d40b06d0 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.837+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d413ad20 con 0x7fd9d40b06d0 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.838+0000 7fd9bb7fe640 10 client.?.objecter ms_handle_connect 0x7fd9d40705f0 2024-03-25T18:28:15.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.838+0000 7fd9bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:15.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.840+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704314936 0 0) 0x7fd9c4001110 con 0x7fd9d40705f0 2024-03-25T18:28:15.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.840+0000 7fd9bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:15.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.840+0000 7fd9bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:15.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.840+0000 7fd9bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:15.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.840+0000 7fd9bb7fe640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:28:15.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.840+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d4143d20 con 0x7fd9d40705f0 2024-03-25T18:28:15.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.841+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438111 0 0) 0x7fd9c0004eb0 con 0x7fd9d4070210 2024-03-25T18:28:15.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.841+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9c4001110 con 0x7fd9d4070210 2024-03-25T18:28:15.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.842+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd9c40047e0 con 0x7fd9d4070210 2024-03-25T18:28:15.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.842+0000 7fd9bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.842+0000 7fd9bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:15.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.842+0000 7fd9bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:15.844 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.842+0000 7fd9bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.843+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 196885376 0 0) 0x7fd9d413ad20 con 0x7fd9d40b06d0 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.843+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9c0004eb0 con 0x7fd9d40b06d0 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.843+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd9cc001c70 con 0x7fd9d40b06d0 2024-03-25T18:28:15.845 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.843+0000 7fd9bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.846 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.843+0000 7fd9bb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:15.846 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.843+0000 7fd9bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:15.846 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.846 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.846 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.846 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.846 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.850 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.850 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.850 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.850 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.850 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.852 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 260925247 0 0) 0x7fd9d4143d20 con 0x7fd9d40705f0 2024-03-25T18:28:15.852 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9d413ad20 con 0x7fd9d40705f0 2024-03-25T18:28:15.852 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd9c80016b0 con 0x7fd9d40705f0 2024-03-25T18:28:15.852 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:15.852 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:15.852 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:15.852 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 407150759 0 0) 0x7fd9c4001110 con 0x7fd9d4070210 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 >> v1:172.21.15.55:6790/0 conn(0x7fd9d40b06d0 legacy=0x7fd9d4143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 >> v1:172.21.15.55:6789/0 conn(0x7fd9d40705f0 legacy=0x7fd9d413b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:15.853 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 1 monclient: found mon.b 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d413c860 con 0x7fd9d4070210 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9d9196640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9d9196640 10 monclient: _renew_subs 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9d9196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9d9196640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d4144470 con 0x7fd9d4070210 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9d9196640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9d9196640 10 monclient: _renew_subs 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9d9196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:15.854 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9d9196640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d413d4b0 con 0x7fd9d4070210 2024-03-25T18:28:15.855 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.845+0000 7fd9d9196640 1 librados: init done 2024-03-25T18:28:15.855 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.846+0000 7fd9d9196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.855 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.846+0000 7fd9d9196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:15.855 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.846+0000 7fd9d9196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:15.855 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:15.846+0000 7fd9d9196640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d413d7b0 con 0x7fd9d4070210 2024-03-25T18:28:15.880 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:15.880+0000 7fa1e3641040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:28:15.961 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:15.959+0000 7f6a26412040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:28:16.021 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:16.020+0000 7fa1e3641040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:28:16.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9c4004bf0 con 0x7fd9d4070210 2024-03-25T18:28:16.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:16.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd9c4004d60 con 0x7fd9d4070210 2024-03-25T18:28:16.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 10 monclient: dump: 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd9c4005250 con 0x7fd9d4070210 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9d159a640 4 set_mon_vals no callback set 2024-03-25T18:28:16.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1819882208 0 0) 0x7fd9c4006760 con 0x7fd9d4070210 2024-03-25T18:28:16.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 10 client.4138.objecter ms_dispatch 0x7fd9d4000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:28:16.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:28:16.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.037+0000 7fd9bb7fe640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:28:16.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.038+0000 7fd9bb7fe640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T18:28:16.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.038+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd9c4006b20 con 0x7fd9d4070210 2024-03-25T18:28:16.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.038+0000 7fd9bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.038+0000 7fd9bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:16.078 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:16.077+0000 7f6a26412040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:28:16.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.111+0000 7fd9d9196640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6d01598c-3cbe-4167-a7e5-606491d1eac8", "id": 3}] 2024-03-25T18:28:16.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.111+0000 7fd9d9196640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6d01598c-3cbe-4167-a7e5-606491d1eac8", "id": 3}] 2024-03-25T18:28:16.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.111+0000 7fd9d9196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.111+0000 7fd9d9196640 1 -- 172.21.15.133:0/784404059 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6d01598c-3cbe-4167-a7e5-606491d1eac8", "id": 3} v 0) v1 -- 0x7fd9d404f730 con 0x7fd9d4070210 2024-03-25T18:28:16.126 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:16.124+0000 7f6a26412040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:28:16.133 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:16.132+0000 7fa1e3641040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:28:16.182 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:16.181+0000 7fa1e3641040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:28:16.219 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.218+0000 7fd9bb7fe640 1 -- 172.21.15.133:0/784404059 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6d01598c-3cbe-4167-a7e5-606491d1eac8", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 661599347 0 525046306) 0x7fd9c40062d0 con 0x7fd9d4070210 2024-03-25T18:28:16.219 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.218+0000 7fd9bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6d01598c-3cbe-4167-a7e5-606491d1eac8", "id": 3}] 2024-03-25T18:28:16.220 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.218+0000 7fd9bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:16.220 INFO:teuthology.orchestra.run.smithi133.stdout:3 2024-03-25T18:28:16.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 10 librados: watch_flush enter 2024-03-25T18:28:16.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 10 librados: watch_flush exit 2024-03-25T18:28:16.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T18:28:16.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 10 client.4138.objecter successfully canceled tick 2024-03-25T18:28:16.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 10 monclient: shutdown 2024-03-25T18:28:16.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:16.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 1 -- 172.21.15.133:0/784404059 >> v1:172.21.15.133:6789/0 conn(0x7fd9d4070210 legacy=0x7fd9d41d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 1 -- 172.21.15.133:0/784404059 shutdown_connections 2024-03-25T18:28:16.221 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 1 -- 172.21.15.133:0/784404059 >> 172.21.15.133:0/784404059 conn(0x7fd9d40b5cd0 msgr2=0x7fd9d40b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:16.222 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 1 -- 172.21.15.133:0/784404059 shutdown_connections 2024-03-25T18:28:16.222 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 1 -- 172.21.15.133:0/784404059 wait complete. 2024-03-25T18:28:16.222 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.219+0000 7fd9d9196640 1 librados: shutdown 2024-03-25T18:28:16.231 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new fb85e8a1-9dab-45ae-8e58-20ea74da1dac 4 2024-03-25T18:28:16.250 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:16.249+0000 7f6a26412040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:28:16.301 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:16.300+0000 7fa1e3641040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:28:16.330 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.329+0000 7fd273e4e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:16.330 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.329+0000 7fd273e4e640 10 monclient: build_initial_monmap 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.329+0000 7fd273e4e640 10 monclient: monmap: 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:28:16.330634+0000 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:28:16.330634+0000 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.331 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.329+0000 7fd273e4e640 1 Processor -- start 2024-03-25T18:28:16.332 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.329+0000 7fd273e4e640 1 -- start start 2024-03-25T18:28:16.332 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.329+0000 7fd273e4e640 10 monclient: init 2024-03-25T18:28:16.332 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd26c0b4700 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd26c142a20 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 10 monclient(hunting): picked mon.noname-a con 0x7fd26c0a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c0a9d60 con 0x7fd26c0a96b0 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c13ff10 con 0x7fd26c0b4700 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c0aa1f0 con 0x7fd26c142a20 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.330+0000 7fd273e4e640 10 monclient(hunting): authenticate will time out at 7329.493323s 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.331+0000 7fd271bea640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fd26c0b4700 0x7fd26c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58280/0 (socket says 172.21.15.133:58280) 2024-03-25T18:28:16.333 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.331+0000 7fd271bea640 1 -- 172.21.15.133:0/2696508098 learned_addr learned my addr 172.21.15.133:0/2696508098 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:16.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016080551 0 0) 0x7fd26c13ff10 con 0x7fd26c0b4700 2024-03-25T18:28:16.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:28:16.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd258003520 con 0x7fd26c0b4700 2024-03-25T18:28:16.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380238378 0 0) 0x7fd26c0a9d60 con 0x7fd26c0a96b0 2024-03-25T18:28:16.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:28:16.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.333+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd26c13ff10 con 0x7fd26c0a96b0 2024-03-25T18:28:16.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.334+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697498130 0 0) 0x7fd26c0aa1f0 con 0x7fd26c142a20 2024-03-25T18:28:16.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.334+0000 7fd270be8640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.334+0000 7fd270be8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.334+0000 7fd270be8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.334+0000 7fd270be8640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:28:16.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.334+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd26c0a9d60 con 0x7fd26c142a20 2024-03-25T18:28:16.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2928824639 0 0) 0x7fd26c13ff10 con 0x7fd26c0a96b0 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 >> v1:172.21.15.55:6790/0 conn(0x7fd26c142a20 legacy=0x7fd26c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 >> v1:172.21.15.133:6789/0 conn(0x7fd26c0b4700 legacy=0x7fd26c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 10 monclient: _finish_hunting 0 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 1 monclient: found mon.noname-a 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c0b4ae0 con 0x7fd26c0a96b0 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 10 monclient: _finish_auth 0 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 10 monclient: _check_auth_tickets 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd2640019f0 con 0x7fd26c0a96b0 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd273e4e640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:28:16.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd273e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd270be8640 10 monclient: dump: 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.336+0000 7fd273e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:16.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.338+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd264001b50 con 0x7fd26c0a96b0 2024-03-25T18:28:16.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.338+0000 7fd270be8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:16.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.338+0000 7fd270be8640 1 -- 172.21.15.133:0/2696508098 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd264001f60 con 0x7fd26c0a96b0 2024-03-25T18:28:16.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.338+0000 7fd270be8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.338+0000 7fd270be8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:16.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.338+0000 7fd270be8640 10 monclient: dump: 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.338+0000 7fd273e4e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.338+0000 7fd273e4e640 4 set_mon_vals no callback set 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.339+0000 7fd273e4e640 10 monclient: shutdown 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.339+0000 7fd273e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:16.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.339+0000 7fd273e4e640 1 -- 172.21.15.133:0/2696508098 >> v1:172.21.15.55:6789/0 conn(0x7fd26c0a96b0 legacy=0x7fd26c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.342 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.339+0000 7fd273e4e640 1 -- 172.21.15.133:0/2696508098 shutdown_connections 2024-03-25T18:28:16.342 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.339+0000 7fd273e4e640 1 -- 172.21.15.133:0/2696508098 >> 172.21.15.133:0/2696508098 conn(0x7fd26c0668e0 msgr2=0x7fd26c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:16.342 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.339+0000 7fd273e4e640 1 -- 172.21.15.133:0/2696508098 shutdown_connections 2024-03-25T18:28:16.342 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.339+0000 7fd273e4e640 1 -- 172.21.15.133:0/2696508098 wait complete. 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient: build_initial_monmap 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient: monmap: 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:16.344 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 1 librados: starting msgr at 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 1 librados: starting objecter 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 1 Processor -- start 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 1 -- start start 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 1 librados: setting wanted keys 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 1 librados: calling monclient init 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient: init 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:16.345 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient(hunting): picked mon.noname-a con 0x7fd26c0a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:16.346 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd26c0b4700 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:16.346 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd26c142a20 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:16.346 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c1395e0 con 0x7fd26c0a96b0 2024-03-25T18:28:16.346 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c138880 con 0x7fd26c0b4700 2024-03-25T18:28:16.346 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c138a10 con 0x7fd26c142a20 2024-03-25T18:28:16.346 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:16.346 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.340+0000 7fd273e4e640 10 monclient(hunting): authenticate will time out at 7329.503680s 2024-03-25T18:28:16.346 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.341+0000 7fd2713e9640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fd26c0b4700 0x7fd26c130fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58286/0 (socket says 172.21.15.133:58286) 2024-03-25T18:28:16.347 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.341+0000 7fd2713e9640 1 -- 172.21.15.133:0/1791859164 learned_addr learned my addr 172.21.15.133:0/1791859164 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:16.347 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.341+0000 7fd2627fc640 10 client.?.objecter ms_handle_connect 0x7fd26c0b4700 2024-03-25T18:28:16.347 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.341+0000 7fd2627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.347 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.341+0000 7fd2627fc640 10 client.?.objecter ms_handle_connect 0x7fd26c0a96b0 2024-03-25T18:28:16.347 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.341+0000 7fd2627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.347 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.341+0000 7fd2627fc640 10 client.?.objecter ms_handle_connect 0x7fd26c142a20 2024-03-25T18:28:16.347 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.341+0000 7fd2627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.348 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79115099 0 0) 0x7fd26c138880 con 0x7fd26c0b4700 2024-03-25T18:28:16.348 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.348 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.348 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.348 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:28:16.348 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd250004eb0 con 0x7fd26c0b4700 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446272106 0 0) 0x7fd26c1395e0 con 0x7fd26c0a96b0 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.343+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd26c138880 con 0x7fd26c0a96b0 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.344+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595341869 0 0) 0x7fd26c138a10 con 0x7fd26c142a20 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.344+0000 7fd2627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.344+0000 7fd2627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.344+0000 7fd2627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.344+0000 7fd2627fc640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:28:16.349 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.344+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd26c1395e0 con 0x7fd26c142a20 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1615270147 0 0) 0x7fd26c138880 con 0x7fd26c0a96b0 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd26c138a10 con 0x7fd26c0a96b0 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd268001950 con 0x7fd26c0a96b0 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 10 monclient(hunting): dump: 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.351 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3843726236 0 0) 0x7fd250004eb0 con 0x7fd26c0b4700 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd26c138880 con 0x7fd26c0b4700 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd25c001a10 con 0x7fd26c0b4700 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.348+0000 7fd2627fc640 10 monclient(hunting): dump: 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.352 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.349+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 651619129 0 0) 0x7fd26c1395e0 con 0x7fd26c142a20 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.349+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd250004eb0 con 0x7fd26c142a20 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.349+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd264001e60 con 0x7fd26c142a20 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.349+0000 7fd2627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.349+0000 7fd2627fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.349+0000 7fd2627fc640 10 monclient(hunting): dump: 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.353 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.354 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.354 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.354 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.354 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.354 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3581058618 0 0) 0x7fd26c138880 con 0x7fd26c0b4700 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 >> v1:172.21.15.55:6790/0 conn(0x7fd26c142a20 legacy=0x7fd26c1d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 >> v1:172.21.15.55:6789/0 conn(0x7fd26c0a96b0 legacy=0x7fd26c138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 1 monclient: found mon.b 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c138ba0 con 0x7fd26c0b4700 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:16.355 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd2627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd273e4e640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd273e4e640 10 monclient: _renew_subs 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd273e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd273e4e640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26c1316b0 con 0x7fd26c0b4700 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd273e4e640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd273e4e640 10 monclient: _renew_subs 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd273e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd273e4e640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26c135ad0 con 0x7fd26c0b4700 2024-03-25T18:28:16.356 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.351+0000 7fd273e4e640 1 librados: init done 2024-03-25T18:28:16.356 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:16.352+0000 7f6a26412040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:28:16.357 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.352+0000 7fd273e4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.357 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.352+0000 7fd273e4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.357 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.352+0000 7fd273e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.357 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.352+0000 7fd273e4e640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c1398c0 con 0x7fd26c0b4700 2024-03-25T18:28:16.404 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:16.403+0000 7fa1e3641040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:28:16.454 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd25c001c70 con 0x7fd26c0b4700 2024-03-25T18:28:16.454 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd25c003a60 con 0x7fd26c0b4700 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 10 monclient: dump: 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd25c004010 con 0x7fd26c0b4700 2024-03-25T18:28:16.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd263fff640 4 set_mon_vals no callback set 2024-03-25T18:28:16.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1233362048 0 0) 0x7fd25c0067d0 con 0x7fd26c0b4700 2024-03-25T18:28:16.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 10 client.4144.objecter ms_dispatch 0x7fd26c000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:28:16.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:28:16.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:28:16.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:28:16.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd25c006b30 con 0x7fd26c0b4700 2024-03-25T18:28:16.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.452+0000 7fd2627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:16.514 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:16.513+0000 7f6a26412040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:28:16.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.525+0000 7fd273e4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fb85e8a1-9dab-45ae-8e58-20ea74da1dac", "id": 4}] 2024-03-25T18:28:16.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.525+0000 7fd273e4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fb85e8a1-9dab-45ae-8e58-20ea74da1dac", "id": 4}] 2024-03-25T18:28:16.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.525+0000 7fd273e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.525+0000 7fd273e4e640 1 -- 172.21.15.133:0/1791859164 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fb85e8a1-9dab-45ae-8e58-20ea74da1dac", "id": 4} v 0) v1 -- 0x7fd26c04f730 con 0x7fd26c0b4700 2024-03-25T18:28:16.558 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:16.557+0000 7f6a26412040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:28:16.569 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:16.569+0000 7fa1e3641040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:28:16.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.589+0000 7fd2627fc640 1 -- 172.21.15.133:0/1791859164 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fb85e8a1-9dab-45ae-8e58-20ea74da1dac", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 787525251 0 1697012583) 0x7fd25c0062c0 con 0x7fd26c0b4700 2024-03-25T18:28:16.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.589+0000 7fd2627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fb85e8a1-9dab-45ae-8e58-20ea74da1dac", "id": 4}] 2024-03-25T18:28:16.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.589+0000 7fd2627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:16.591 INFO:teuthology.orchestra.run.smithi133.stdout:4 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 10 librados: watch_flush enter 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 10 librados: watch_flush exit 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 10 monclient: shutdown 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 1 -- 172.21.15.133:0/1791859164 >> v1:172.21.15.133:6789/0 conn(0x7fd26c0b4700 legacy=0x7fd26c130fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 1 -- 172.21.15.133:0/1791859164 shutdown_connections 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 1 -- 172.21.15.133:0/1791859164 >> 172.21.15.133:0/1791859164 conn(0x7fd26c0668e0 msgr2=0x7fd26c0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 1 -- 172.21.15.133:0/1791859164 shutdown_connections 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 1 -- 172.21.15.133:0/1791859164 wait complete. 2024-03-25T18:28:16.592 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.590+0000 7fd273e4e640 1 librados: shutdown 2024-03-25T18:28:16.601 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new f181ab39-94f0-43d3-a755-cc9640c4f2e3 5 2024-03-25T18:28:16.616 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:16.615+0000 7fa1e3641040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:28:16.721 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:16.720+0000 7f6a26412040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:28:16.743 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.742+0000 7f1c3aa4a640 10 monclient: get_monmap_and_config 2024-03-25T18:28:16.743 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.742+0000 7f1c3aa4a640 10 monclient: build_initial_monmap 2024-03-25T18:28:16.743 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.742+0000 7f1c3aa4a640 10 monclient: monmap: 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:28:16.743705+0000 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:28:16.743705+0000 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:16.744 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.745 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.742+0000 7f1c3aa4a640 1 Processor -- start 2024-03-25T18:28:16.745 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 1 -- start start 2024-03-25T18:28:16.745 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 10 monclient: init 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 10 monclient(hunting): picked mon.noname-b con 0x7f1c340b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 10 monclient(hunting): picked mon.noname-a con 0x7f1c34142a20 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 10 monclient(hunting): picked mon.noname-c con 0x7f1c340a96b0 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340a9d60 con 0x7f1c34142a20 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3413ff10 con 0x7f1c340a96b0 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340aa1f0 con 0x7f1c340b4700 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.743+0000 7f1c3aa4a640 10 monclient(hunting): authenticate will time out at 7329.906555s 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.744+0000 7f1c38fe7640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f1c340a96b0 0x7f1c3413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58292/0 (socket says 172.21.15.133:58292) 2024-03-25T18:28:16.746 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.744+0000 7f1c38fe7640 1 -- 172.21.15.133:0/1293166873 learned_addr learned my addr 172.21.15.133:0/1293166873 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:16.747 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723900994 0 0) 0x7f1c3413ff10 con 0x7f1c340a96b0 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c1c003520 con 0x7f1c340a96b0 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893464009 0 0) 0x7f1c340a9d60 con 0x7f1c34142a20 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:28:16.748 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.746+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3413ff10 con 0x7f1c34142a20 2024-03-25T18:28:16.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.747+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063299992 0 0) 0x7f1c340aa1f0 con 0x7f1c340b4700 2024-03-25T18:28:16.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.747+0000 7f1c32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.747+0000 7f1c32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.747+0000 7f1c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.747+0000 7f1c32ffd640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:28:16.749 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.747+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c340a9d60 con 0x7f1c340b4700 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3439168178 0 0) 0x7f1c1c003520 con 0x7f1c340a96b0 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 >> v1:172.21.15.55:6790/0 conn(0x7f1c340b4700 legacy=0x7f1c34142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 >> v1:172.21.15.55:6789/0 conn(0x7f1c34142a20 legacy=0x7f1c3413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 1 monclient: found mon.noname-c 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c340b4ae0 con 0x7f1c340a96b0 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:16.751 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c2c001a10 con 0x7f1c340a96b0 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c3aa4a640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c32ffd640 10 monclient: dump: 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.752 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.749+0000 7f1c3aa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:16.754 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c2c001b70 con 0x7f1c340a96b0 2024-03-25T18:28:16.754 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:16.754 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c32ffd640 1 -- 172.21.15.133:0/1293166873 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c2c001f80 con 0x7f1c340a96b0 2024-03-25T18:28:16.754 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c3aa4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:16.754 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c3aa4a640 4 set_mon_vals no callback set 2024-03-25T18:28:16.754 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.754 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:16.754 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c32ffd640 10 monclient: dump: 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c3aa4a640 10 monclient: shutdown 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c3aa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/1293166873 >> v1:172.21.15.133:6789/0 conn(0x7f1c340a96b0 legacy=0x7f1c3413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.755 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/1293166873 shutdown_connections 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/1293166873 >> 172.21.15.133:0/1293166873 conn(0x7f1c340668e0 msgr2=0x7f1c34066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/1293166873 shutdown_connections 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.751+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/1293166873 wait complete. 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.752+0000 7f1c3aa4a640 10 monclient: build_initial_monmap 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.752+0000 7f1c3aa4a640 10 monclient: monmap: 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:16.756 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.757 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.752+0000 7f1c3aa4a640 1 librados: starting msgr at 2024-03-25T18:28:16.757 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.752+0000 7f1c3aa4a640 1 librados: starting objecter 2024-03-25T18:28:16.757 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.752+0000 7f1c3aa4a640 1 Processor -- start 2024-03-25T18:28:16.757 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 1 -- start start 2024-03-25T18:28:16.757 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 1 librados: setting wanted keys 2024-03-25T18:28:16.757 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 1 librados: calling monclient init 2024-03-25T18:28:16.757 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 10 monclient: init 2024-03-25T18:28:16.759 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:16.759 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:16.759 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 10 monclient(hunting): picked mon.noname-a con 0x7f1c340a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:16.759 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 10 monclient(hunting): picked mon.noname-c con 0x7f1c340b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:16.759 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 10 monclient(hunting): picked mon.noname-b con 0x7f1c34142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:16.759 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3413b110 con 0x7f1c340a96b0 2024-03-25T18:28:16.759 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c341dbe60 con 0x7f1c34142a20 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340abc90 con 0x7f1c340b4700 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.753+0000 7f1c3aa4a640 10 monclient(hunting): authenticate will time out at 7329.916414s 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.754+0000 7f1c38fe7640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f1c34142a20 0x7f1c341db750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58306/0 (socket says 172.21.15.133:58306) 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.754+0000 7f1c38fe7640 1 -- 172.21.15.133:0/2462377997 learned_addr learned my addr 172.21.15.133:0/2462377997 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.754+0000 7f1c30ff9640 10 client.?.objecter ms_handle_connect 0x7f1c34142a20 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.754+0000 7f1c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.754+0000 7f1c30ff9640 10 client.?.objecter ms_handle_connect 0x7f1c340a96b0 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.754+0000 7f1c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.754+0000 7f1c30ff9640 10 client.?.objecter ms_handle_connect 0x7f1c340b4700 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.754+0000 7f1c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289836592 0 0) 0x7f1c341dbe60 con 0x7f1c34142a20 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c20004eb0 con 0x7f1c34142a20 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696508654 0 0) 0x7f1c3413b110 con 0x7f1c340a96b0 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c341dbe60 con 0x7f1c340a96b0 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464566601 0 0) 0x7f1c340abc90 con 0x7f1c340b4700 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.761 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:28:16.762 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.756+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3413b110 con 0x7f1c340b4700 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.760+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1291564788 0 0) 0x7f1c20004eb0 con 0x7f1c34142a20 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.760+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c340abc90 con 0x7f1c34142a20 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.760+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c2c001f60 con 0x7f1c34142a20 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.760+0000 7f1c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.760+0000 7f1c30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.760+0000 7f1c30ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.763 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.760+0000 7f1c30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.761+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 19939406 0 0) 0x7f1c341dbe60 con 0x7f1c340a96b0 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.761+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c20004eb0 con 0x7f1c340a96b0 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.761+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c28001870 con 0x7f1c340a96b0 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.761+0000 7f1c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.761+0000 7f1c30ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.761+0000 7f1c30ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.764 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.765 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.765 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.765 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.765 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.765 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.765 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.765 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.762+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4002246458 0 0) 0x7f1c3413b110 con 0x7f1c340b4700 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.762+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c341dbe60 con 0x7f1c340b4700 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.762+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c240018f0 con 0x7f1c340b4700 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.762+0000 7f1c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.762+0000 7f1c30ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.762+0000 7f1c30ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.767 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.763+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2575636983 0 0) 0x7f1c340abc90 con 0x7f1c34142a20 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.763+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 >> v1:172.21.15.55:6790/0 conn(0x7f1c340b4700 legacy=0x7f1c341397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.763+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 >> v1:172.21.15.55:6789/0 conn(0x7f1c340a96b0 legacy=0x7f1c341db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.763+0000 7f1c30ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.763+0000 7f1c30ff9640 1 monclient: found mon.b 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c30ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c340abf70 con 0x7f1c34142a20 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c30ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:16.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c30ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c3aa4a640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c3aa4a640 10 monclient: _renew_subs 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c3aa4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c340ac250 con 0x7f1c34142a20 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c3aa4a640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c3aa4a640 10 monclient: _renew_subs 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c3aa4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3414e6c0 con 0x7f1c34142a20 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.764+0000 7f1c3aa4a640 1 librados: init done 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.765+0000 7f1c3aa4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.765+0000 7f1c3aa4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.765+0000 7f1c3aa4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.765+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3404f730 con 0x7f1c34142a20 2024-03-25T18:28:16.769 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.765+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c2c0020c0 con 0x7f1c34142a20 2024-03-25T18:28:16.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.765+0000 7f1c30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:16.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.765+0000 7f1c327fc640 4 set_mon_vals no callback set 2024-03-25T18:28:16.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c2c004fb0 con 0x7f1c34142a20 2024-03-25T18:28:16.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 10 monclient: dump: 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1c2c0054a0 con 0x7f1c34142a20 2024-03-25T18:28:16.771 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 476011154 0 0) 0x7f1c2c006ca0 con 0x7f1c34142a20 2024-03-25T18:28:16.772 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 10 client.4150.objecter ms_dispatch 0x7f1c34000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:28:16.772 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:28:16.772 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:28:16.772 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.767+0000 7f1c30ff9640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T18:28:16.780 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:16.779+0000 7fa1e3641040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:28:16.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.863+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1c2c0067d0 con 0x7f1c34142a20 2024-03-25T18:28:16.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.863+0000 7f1c30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.863+0000 7f1c30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:16.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.935+0000 7f1c3aa4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f181ab39-94f0-43d3-a755-cc9640c4f2e3", "id": 5}] 2024-03-25T18:28:16.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.935+0000 7f1c3aa4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f181ab39-94f0-43d3-a755-cc9640c4f2e3", "id": 5}] 2024-03-25T18:28:16.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.935+0000 7f1c3aa4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:16.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.935+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/2462377997 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f181ab39-94f0-43d3-a755-cc9640c4f2e3", "id": 5} v 0) v1 -- 0x7f1c3413b2a0 con 0x7f1c34142a20 2024-03-25T18:28:16.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.993+0000 7f1c30ff9640 1 -- 172.21.15.133:0/2462377997 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f181ab39-94f0-43d3-a755-cc9640c4f2e3", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3050142505 0 1988412176) 0x7f1c2c005750 con 0x7f1c34142a20 2024-03-25T18:28:16.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.993+0000 7f1c30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f181ab39-94f0-43d3-a755-cc9640c4f2e3", "id": 5}] 2024-03-25T18:28:16.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.993+0000 7f1c30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:16.995 INFO:teuthology.orchestra.run.smithi133.stdout:5 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.993+0000 7f1c3aa4a640 10 librados: watch_flush enter 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.993+0000 7f1c3aa4a640 10 librados: watch_flush exit 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.993+0000 7f1c3aa4a640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.993+0000 7f1c3aa4a640 10 client.4150.objecter successfully canceled tick 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.994+0000 7f1c3aa4a640 10 monclient: shutdown 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.994+0000 7f1c3aa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.994+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/2462377997 >> v1:172.21.15.133:6789/0 conn(0x7f1c34142a20 legacy=0x7f1c341db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.994+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/2462377997 shutdown_connections 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.994+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/2462377997 >> 172.21.15.133:0/2462377997 conn(0x7f1c340668e0 msgr2=0x7f1c3413e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.994+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/2462377997 shutdown_connections 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.994+0000 7f1c3aa4a640 1 -- 172.21.15.133:0/2462377997 wait complete. 2024-03-25T18:28:16.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:16.994+0000 7f1c3aa4a640 1 librados: shutdown 2024-03-25T18:28:17.005 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac 6 2024-03-25T18:28:17.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.104+0000 7f2a21874640 10 monclient: get_monmap_and_config 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.104+0000 7f2a21874640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.104+0000 7f2a21874640 10 monclient: monmap: 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:28:17.106214+0000 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:28:17.106214+0000 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:17.107 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.105+0000 7f2a21874640 1 Processor -- start 2024-03-25T18:28:17.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.105+0000 7f2a21874640 1 -- start start 2024-03-25T18:28:17.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.105+0000 7f2a21874640 10 monclient: init 2024-03-25T18:28:17.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.105+0000 7f2a21874640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.105+0000 7f2a21874640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:17.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.105+0000 7f2a21874640 10 monclient(hunting): picked mon.noname-a con 0x7f2a1c0b4700 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:17.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.105+0000 7f2a21874640 10 monclient(hunting): picked mon.noname-c con 0x7f2a1c142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:17.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.105+0000 7f2a21874640 10 monclient(hunting): picked mon.noname-b con 0x7f2a1c0a96b0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:17.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.105+0000 7f2a21874640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0a9d60 con 0x7f2a1c0b4700 2024-03-25T18:28:17.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.106+0000 7f2a21874640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c13ff10 con 0x7f2a1c142a20 2024-03-25T18:28:17.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.106+0000 7f2a21874640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0aa1f0 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.106+0000 7f2a21874640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.106+0000 7f2a21874640 10 monclient(hunting): authenticate will time out at 7330.268734s 2024-03-25T18:28:17.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.106+0000 7f2a1a7fc640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2a1c142a20 0x7f2a1c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58318/0 (socket says 172.21.15.133:58318) 2024-03-25T18:28:17.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.106+0000 7f2a1a7fc640 1 -- 172.21.15.133:0/2280898860 learned_addr learned my addr 172.21.15.133:0/2280898860 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.108+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676289999 0 0) 0x7f2a1c13ff10 con 0x7f2a1c142a20 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.108+0000 7f2a19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.108+0000 7f2a19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.108+0000 7f2a19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.108+0000 7f2a19ffb640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.108+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a04003520 con 0x7f2a1c142a20 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125214374 0 0) 0x7f2a1c0a9d60 con 0x7f2a1c0b4700 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.111 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:28:17.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a1c13ff10 con 0x7f2a1c0b4700 2024-03-25T18:28:17.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575493331 0 0) 0x7f2a1c0aa1f0 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:28:17.112 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.109+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a1c0a9d60 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735398111 0 0) 0x7f2a04003520 con 0x7f2a1c142a20 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 >> v1:172.21.15.55:6790/0 conn(0x7f2a1c0a96b0 legacy=0x7f2a1c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 >> v1:172.21.15.55:6789/0 conn(0x7f2a1c0b4700 legacy=0x7f2a1c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 1 monclient: found mon.noname-c 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c0b4ae0 con 0x7f2a1c142a20 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.113 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.114 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a21874640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:28:17.114 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.111+0000 7f2a21874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:17.114 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.112+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a0c001650 con 0x7f2a1c142a20 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.112+0000 7f2a19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.112+0000 7f2a19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.112+0000 7f2a19ffb640 10 monclient: dump: 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.115 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.112+0000 7f2a21874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.113+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a0c0018a0 con 0x7f2a1c142a20 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.113+0000 7f2a19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.113+0000 7f2a19ffb640 1 -- 172.21.15.133:0/2280898860 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a0c001c30 con 0x7f2a1c142a20 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.113+0000 7f2a21874640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.113+0000 7f2a21874640 4 set_mon_vals no callback set 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.113+0000 7f2a19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.113+0000 7f2a19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.113+0000 7f2a19ffb640 10 monclient: dump: 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.116 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.117 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.117 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.117 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.117 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.120 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.114+0000 7f2a21874640 10 monclient: shutdown 2024-03-25T18:28:17.120 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.114+0000 7f2a21874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:17.120 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.114+0000 7f2a21874640 1 -- 172.21.15.133:0/2280898860 >> v1:172.21.15.133:6789/0 conn(0x7f2a1c142a20 legacy=0x7f2a1c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.120 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.114+0000 7f2a21874640 1 -- 172.21.15.133:0/2280898860 shutdown_connections 2024-03-25T18:28:17.121 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.114+0000 7f2a21874640 1 -- 172.21.15.133:0/2280898860 >> 172.21.15.133:0/2280898860 conn(0x7f2a1c0668e0 msgr2=0x7f2a1c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:17.121 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.114+0000 7f2a21874640 1 -- 172.21.15.133:0/2280898860 shutdown_connections 2024-03-25T18:28:17.121 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.114+0000 7f2a21874640 1 -- 172.21.15.133:0/2280898860 wait complete. 2024-03-25T18:28:17.121 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.114+0000 7f2a21874640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.121 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.114+0000 7f2a21874640 10 monclient: monmap: 2024-03-25T18:28:17.121 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:17.121 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:17.121 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.115+0000 7f2a21874640 1 librados: starting msgr at 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.115+0000 7f2a21874640 1 librados: starting objecter 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.115+0000 7f2a21874640 1 Processor -- start 2024-03-25T18:28:17.122 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.115+0000 7f2a21874640 1 -- start start 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.115+0000 7f2a21874640 1 librados: setting wanted keys 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.115+0000 7f2a21874640 1 librados: calling monclient init 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.115+0000 7f2a21874640 10 monclient: init 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 10 monclient(hunting): picked mon.noname-a con 0x7f2a1c0a96b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 10 monclient(hunting): picked mon.noname-c con 0x7f2a1c0b4700 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 10 monclient(hunting): picked mon.noname-b con 0x7f2a1c142a20 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0ad430 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0b1450 con 0x7f2a1c142a20 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0b15e0 con 0x7f2a1c0b4700 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.123 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a21874640 10 monclient(hunting): authenticate will time out at 7330.278841s 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a1b7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2a1c142a20 0x7f2a1c0b0d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58334/0 (socket says 172.21.15.133:58334) 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a1b7fe640 1 -- 172.21.15.133:0/3158845486 learned_addr learned my addr 172.21.15.133:0/3158845486 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a03fff640 10 client.?.objecter ms_handle_connect 0x7f2a1c142a20 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a03fff640 10 client.?.objecter ms_handle_connect 0x7f2a1c0a96b0 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.116+0000 7f2a03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.117+0000 7f2a03fff640 10 client.?.objecter ms_handle_connect 0x7f2a1c0b4700 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.117+0000 7f2a03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.118+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628373020 0 0) 0x7f2a1c0b1450 con 0x7f2a1c142a20 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.118+0000 7f2a03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.118+0000 7f2a03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.124 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.118+0000 7f2a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.118+0000 7f2a03fff640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.118+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a08004eb0 con 0x7f2a1c142a20 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564858862 0 0) 0x7f2a1c0ad430 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a1c0b1450 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149984869 0 0) 0x7f2a1c0b15e0 con 0x7f2a1c0b4700 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:28:17.125 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.119+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a1c0ad430 con 0x7f2a1c0b4700 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335855169 0 0) 0x7f2a08004eb0 con 0x7f2a1c142a20 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a1c0b15e0 con 0x7f2a1c142a20 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a14001d30 con 0x7f2a1c142a20 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 10 monclient(hunting): dump: 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.128 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1013278853 0 0) 0x7f2a1c0b1450 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a08004eb0 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a10001930 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.123+0000 7f2a03fff640 10 monclient(hunting): dump: 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.130 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.125+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399635236 0 0) 0x7f2a1c0ad430 con 0x7f2a1c0b4700 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.125+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a1c0b1450 con 0x7f2a1c0b4700 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.125+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a0c001a10 con 0x7f2a1c0b4700 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.125+0000 7f2a03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.125+0000 7f2a03fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.125+0000 7f2a03fff640 10 monclient(hunting): dump: 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.131 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3339649695 0 0) 0x7f2a08004eb0 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 >> v1:172.21.15.55:6790/0 conn(0x7f2a1c0b4700 legacy=0x7f2a1c13b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 >> v1:172.21.15.133:6789/0 conn(0x7f2a1c142a20 legacy=0x7f2a1c0b0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 1 monclient: found mon.a 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c0b1770 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a21874640 5 monclient: authenticate success, global_id 4152 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a21874640 10 monclient: _renew_subs 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a21874640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a21874640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c1342f0 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a21874640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a21874640 10 monclient: _renew_subs 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a21874640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a21874640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c134820 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.126+0000 7f2a21874640 1 librados: init done 2024-03-25T18:28:17.134 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.127+0000 7f2a21874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.134 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.127+0000 7f2a21874640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.134 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.127+0000 7f2a21874640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:17.134 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.127+0000 7f2a21874640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1c134140 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a10001d40 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a10003790 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 10 monclient: dump: 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.234 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2a10003d00 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a197fa640 4 set_mon_vals no callback set 2024-03-25T18:28:17.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2745697598 0 0) 0x7f2a100054b0 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 10 client.4152.objecter ms_dispatch 0x7f2a1c000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:28:17.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 3 client.4152.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:28:17.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 3 client.4152.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:28:17.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 20 client.4152.objecter dump_active .. 0 homeless 2024-03-25T18:28:17.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2a10004ec0 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.233+0000 7f2a03fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:17.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.302+0000 7f2a21874640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac", "id": 6}] 2024-03-25T18:28:17.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.302+0000 7f2a21874640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac", "id": 6}] 2024-03-25T18:28:17.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.302+0000 7f2a21874640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:17.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.302+0000 7f2a21874640 1 -- 172.21.15.133:0/3158845486 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac", "id": 6} v 0) v1 -- 0x7f2a1c13be10 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.332 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.331+0000 7f2a03fff640 1 -- 172.21.15.133:0/3158845486 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 729659653 0 1113811849) 0x7f2a100051d0 con 0x7f2a1c0a96b0 2024-03-25T18:28:17.332 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.331+0000 7f2a03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac", "id": 6}] 2024-03-25T18:28:17.332 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.331+0000 7f2a03fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:17.332 INFO:teuthology.orchestra.run.smithi133.stdout:6 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.331+0000 7f2a21874640 10 librados: watch_flush enter 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.331+0000 7f2a21874640 10 librados: watch_flush exit 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.331+0000 7f2a21874640 20 client.4152.objecter shutdown clearing up homeless session... 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.331+0000 7f2a21874640 10 client.4152.objecter successfully canceled tick 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.332+0000 7f2a21874640 10 monclient: shutdown 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.332+0000 7f2a21874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.332+0000 7f2a21874640 1 -- 172.21.15.133:0/3158845486 >> v1:172.21.15.55:6789/0 conn(0x7f2a1c0a96b0 legacy=0x7f2a1c0abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.332+0000 7f2a21874640 1 -- 172.21.15.133:0/3158845486 shutdown_connections 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.332+0000 7f2a21874640 1 -- 172.21.15.133:0/3158845486 >> 172.21.15.133:0/3158845486 conn(0x7f2a1c0668e0 msgr2=0x7f2a1c1364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:17.334 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.332+0000 7f2a21874640 1 -- 172.21.15.133:0/3158845486 shutdown_connections 2024-03-25T18:28:17.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.332+0000 7f2a21874640 1 -- 172.21.15.133:0/3158845486 wait complete. 2024-03-25T18:28:17.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.332+0000 7f2a21874640 1 librados: shutdown 2024-03-25T18:28:17.343 DEBUG:teuthology.orchestra.run.smithi133:> sudo ceph --cluster ceph osd new 11e302d1-2e5b-46e5-9ef8-09c7b65c028a 7 2024-03-25T18:28:17.441 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.439+0000 7fafe6723640 10 monclient: get_monmap_and_config 2024-03-25T18:28:17.441 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.439+0000 7fafe6723640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.441 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.439+0000 7fafe6723640 10 monclient: monmap: 2024-03-25T18:28:17.441 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:17.441 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.441 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:28:17.441236+0000 2024-03-25T18:28:17.441 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:28:17.441236+0000 2024-03-25T18:28:17.442 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.442 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.442 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:17.442 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:17.442 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:17.442 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.440+0000 7fafe6723640 1 Processor -- start 2024-03-25T18:28:17.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.440+0000 7fafe6723640 1 -- start start 2024-03-25T18:28:17.443 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.440+0000 7fafe6723640 10 monclient: init 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 10 monclient(hunting): picked mon.noname-b con 0x7fafe00668e0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 10 monclient(hunting): picked mon.noname-c con 0x7fafe0066cc0 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 10 monclient(hunting): picked mon.noname-a con 0x7fafe0145cf0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0146340 con 0x7fafe0145cf0 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0142dc0 con 0x7fafe0066cc0 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe006f450 con 0x7fafe00668e0 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafe6723640 10 monclient(hunting): authenticate will time out at 7330.603936s 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafdf7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fafe0066cc0 0x7fafe01422b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58348/0 (socket says 172.21.15.133:58348) 2024-03-25T18:28:17.444 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.441+0000 7fafdf7fe640 1 -- 172.21.15.133:0/169442261 learned_addr learned my addr 172.21.15.133:0/169442261 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:17.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.443+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096803495 0 0) 0x7fafe0142dc0 con 0x7fafe0066cc0 2024-03-25T18:28:17.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.443+0000 7fafdeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.443+0000 7fafdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.443+0000 7fafdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.443+0000 7fafdeffd640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:28:17.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.443+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc8003520 con 0x7fafe0066cc0 2024-03-25T18:28:17.446 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204948987 0 0) 0x7fafe006f450 con 0x7fafe00668e0 2024-03-25T18:28:17.446 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.446 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.446 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.446 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:28:17.446 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe0142dc0 con 0x7fafe00668e0 2024-03-25T18:28:17.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546800297 0 0) 0x7fafe0146340 con 0x7fafe0145cf0 2024-03-25T18:28:17.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:28:17.447 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.444+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe006f450 con 0x7fafe0145cf0 2024-03-25T18:28:17.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 918372694 0 0) 0x7fafc8003520 con 0x7fafe0066cc0 2024-03-25T18:28:17.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 >> v1:172.21.15.55:6790/0 conn(0x7fafe00668e0 legacy=0x7fafe013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 >> v1:172.21.15.55:6789/0 conn(0x7fafe0145cf0 legacy=0x7fafe014dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 1 monclient: found mon.noname-c 2024-03-25T18:28:17.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.448 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe006f730 con 0x7fafe0066cc0 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafe6723640 5 monclient: authenticate success, global_id 4159 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.446+0000 7fafe6723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.447+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fafd4001650 con 0x7fafe0066cc0 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.447+0000 7fafdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.447+0000 7fafdeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.447+0000 7fafdeffd640 10 monclient: dump: 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.449 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.450 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.450 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.450 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.450 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.450 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.450 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.447+0000 7fafe6723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:17.454 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.448+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafd4001830 con 0x7fafe0066cc0 2024-03-25T18:28:17.454 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.448+0000 7fafdeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:17.454 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.448+0000 7fafdeffd640 1 -- 172.21.15.133:0/169442261 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fafd4001bc0 con 0x7fafe0066cc0 2024-03-25T18:28:17.454 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.448+0000 7fafdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.454 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.448+0000 7fafdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:17.454 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.448+0000 7fafdeffd640 10 monclient: dump: 2024-03-25T18:28:17.454 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.454 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.448+0000 7fafe6723640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.448+0000 7fafe6723640 4 set_mon_vals no callback set 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.449+0000 7fafe6723640 10 monclient: shutdown 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.449+0000 7fafe6723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:17.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.449+0000 7fafe6723640 1 -- 172.21.15.133:0/169442261 >> v1:172.21.15.133:6789/0 conn(0x7fafe0066cc0 legacy=0x7fafe01422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.449+0000 7fafe6723640 1 -- 172.21.15.133:0/169442261 shutdown_connections 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.449+0000 7fafe6723640 1 -- 172.21.15.133:0/169442261 >> 172.21.15.133:0/169442261 conn(0x7fafe006aba0 msgr2=0x7fafe006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.449+0000 7fafe6723640 1 -- 172.21.15.133:0/169442261 shutdown_connections 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.449+0000 7fafe6723640 1 -- 172.21.15.133:0/169442261 wait complete. 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.450+0000 7fafe6723640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.450+0000 7fafe6723640 10 monclient: monmap: 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 0 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 0.000000 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:created 0.000000 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 1 2024-03-25T18:28:17.456 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.450+0000 7fafe6723640 1 librados: starting msgr at 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.450+0000 7fafe6723640 1 librados: starting objecter 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.450+0000 7fafe6723640 1 Processor -- start 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.450+0000 7fafe6723640 1 -- start start 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.450+0000 7fafe6723640 1 librados: setting wanted keys 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.450+0000 7fafe6723640 1 librados: calling monclient init 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.450+0000 7fafe6723640 10 monclient: init 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 10 monclient(hunting): picked mon.noname-b con 0x7fafe00668e0 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 10 monclient(hunting): picked mon.noname-c con 0x7fafe0145cf0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 10 monclient(hunting): picked mon.noname-a con 0x7fafe013dd90 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:17.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0120800 con 0x7fafe013dd90 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0120ae0 con 0x7fafe00668e0 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe011f660 con 0x7fafe0145cf0 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafe6723640 10 monclient(hunting): authenticate will time out at 7330.614069s 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafdffff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fafe00668e0 0x7fafe011e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.133:58360/0 (socket says 172.21.15.133:58360) 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafdffff640 1 -- 172.21.15.133:0/3040647901 learned_addr learned my addr 172.21.15.133:0/3040647901 (peer_addr_for_me v1:172.21.15.133:0/0) 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafdcff9640 10 client.?.objecter ms_handle_connect 0x7fafe00668e0 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.451+0000 7fafdcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.452+0000 7fafdcff9640 10 client.?.objecter ms_handle_connect 0x7fafe013dd90 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.452+0000 7fafdcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.452+0000 7fafdcff9640 10 client.?.objecter ms_handle_connect 0x7fafe0145cf0 2024-03-25T18:28:17.458 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.452+0000 7fafdcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.453+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005662434 0 0) 0x7fafe0120ae0 con 0x7fafe00668e0 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.453+0000 7fafdcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.453+0000 7fafdcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.453+0000 7fafdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.453+0000 7fafdcff9640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.453+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafcc004eb0 con 0x7fafe00668e0 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.454+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814623591 0 0) 0x7fafe011f660 con 0x7fafe0145cf0 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.454+0000 7fafdcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.454+0000 7fafdcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.454+0000 7fafdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.454+0000 7fafdcff9640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.454+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe0120ae0 con 0x7fafe0145cf0 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.455+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458696679 0 0) 0x7fafe0120800 con 0x7fafe013dd90 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.455+0000 7fafdcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.455+0000 7fafdcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.455+0000 7fafdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.455+0000 7fafdcff9640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.455+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe011f660 con 0x7fafe013dd90 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2901878102 0 0) 0x7fafcc004eb0 con 0x7fafe00668e0 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafe0120800 con 0x7fafe00668e0 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fafd0001900 con 0x7fafe00668e0 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 10 monclient(hunting): dump: 2024-03-25T18:28:17.462 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 881626825 0 0) 0x7fafe0120ae0 con 0x7fafe0145cf0 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafcc004eb0 con 0x7fafe0145cf0 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fafd4001b40 con 0x7fafe0145cf0 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.458+0000 7fafdcff9640 10 monclient(hunting): dump: 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.464 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.460+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2245859027 0 0) 0x7fafe0120800 con 0x7fafe00668e0 2024-03-25T18:28:17.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 >> v1:172.21.15.55:6790/0 conn(0x7fafe0145cf0 legacy=0x7fafe013d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 >> v1:172.21.15.55:6789/0 conn(0x7fafe013dd90 legacy=0x7fafe01d7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 1 monclient: found mon.b 2024-03-25T18:28:17.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:17.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe013e290 con 0x7fafe00668e0 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 5 monclient: authenticate success, global_id 4162 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 10 monclient: _renew_subs 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe01d7930 con 0x7fafe00668e0 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 10 monclient: _renew_subs 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe01d7eb0 con 0x7fafe00668e0 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 1 librados: init done 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:17.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.461+0000 7fafe6723640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe011f7f0 con 0x7fafe00668e0 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafd0003610 con 0x7fafe00668e0 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fafd00039e0 con 0x7fafe00668e0 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 10 monclient: dump: 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:epoch 1 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.559 INFO:teuthology.orchestra.run.smithi133.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:election_strategy: 3 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fafd0003f90 con 0x7fafe00668e0 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafde7fc640 4 set_mon_vals no callback set 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1727862308 0 0) 0x7fafd00066e0 con 0x7fafe00668e0 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 10 client.4162.objecter ms_dispatch 0x7fafe0000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fafd0006ac0 con 0x7fafe00668e0 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.560 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.557+0000 7fafdcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:17.634 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.633+0000 7fafe6723640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "11e302d1-2e5b-46e5-9ef8-09c7b65c028a", "id": 7}] 2024-03-25T18:28:17.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.633+0000 7fafe6723640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "11e302d1-2e5b-46e5-9ef8-09c7b65c028a", "id": 7}] 2024-03-25T18:28:17.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.633+0000 7fafe6723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:17.635 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.633+0000 7fafe6723640 1 -- 172.21.15.133:0/3040647901 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd new", "uuid": "11e302d1-2e5b-46e5-9ef8-09c7b65c028a", "id": 7} v 0) v1 -- 0x7fafe011f470 con 0x7fafe00668e0 2024-03-25T18:28:17.649 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:17.648+0000 7fa1e3641040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:28:17.698 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.697+0000 7fafdcff9640 1 -- 172.21.15.133:0/3040647901 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "11e302d1-2e5b-46e5-9ef8-09c7b65c028a", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3334563146 0 1371665406) 0x7fafd0004240 con 0x7fafe00668e0 2024-03-25T18:28:17.698 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.697+0000 7fafdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "11e302d1-2e5b-46e5-9ef8-09c7b65c028a", "id": 7}] 2024-03-25T18:28:17.699 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.697+0000 7fafdcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:17.699 INFO:teuthology.orchestra.run.smithi133.stdout:7 2024-03-25T18:28:17.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.697+0000 7fafe6723640 10 librados: watch_flush enter 2024-03-25T18:28:17.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.697+0000 7fafe6723640 10 librados: watch_flush exit 2024-03-25T18:28:17.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.697+0000 7fafe6723640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T18:28:17.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.697+0000 7fafe6723640 10 client.4162.objecter successfully canceled tick 2024-03-25T18:28:17.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.698+0000 7fafe6723640 10 monclient: shutdown 2024-03-25T18:28:17.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.698+0000 7fafe6723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:17.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.698+0000 7fafe6723640 1 -- 172.21.15.133:0/3040647901 >> v1:172.21.15.133:6789/0 conn(0x7fafe00668e0 legacy=0x7fafe011e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.698+0000 7fafe6723640 1 -- 172.21.15.133:0/3040647901 shutdown_connections 2024-03-25T18:28:17.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.698+0000 7fafe6723640 1 -- 172.21.15.133:0/3040647901 >> 172.21.15.133:0/3040647901 conn(0x7fafe006aba0 msgr2=0x7fafe00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:17.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.698+0000 7fafe6723640 1 -- 172.21.15.133:0/3040647901 shutdown_connections 2024-03-25T18:28:17.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.698+0000 7fafe6723640 1 -- 172.21.15.133:0/3040647901 wait complete. 2024-03-25T18:28:17.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-25T18:28:17.698+0000 7fafe6723640 1 librados: shutdown 2024-03-25T18:28:17.711 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:28:17.711 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:28:17.711 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:28:17.714 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:28:17.714 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:28:17.714 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:28:17.714 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:28:17.716 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:28:17.716 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:28:17.716 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:28:17.716 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:28:17.718 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:28:17.718 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:28:17.718 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:28:17.719 DEBUG:teuthology.orchestra.run.smithi055:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:28:17.721 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:28:17.721 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:28:17.721 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:28:17.721 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:28:17.723 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:28:17.723 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:28:17.723 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:28:17.723 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:28:17.725 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:28:17.725 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:28:17.726 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:28:17.726 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:28:17.728 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:28:17.728 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:28:17.728 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:28:17.728 DEBUG:teuthology.orchestra.run.smithi133:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:28:17.730 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:28:17.731 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:17.855 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.854+0000 7fc25923e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.854+0000 7fc25923e640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.854+0000 7fc25923e640 10 monclient: monmap: 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 0 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:28:17.855203+0000 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:28:17.855203+0000 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:17.856 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:17.857 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:17.857 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:17.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 1 Processor -- start 2024-03-25T18:28:17.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 1 -- start start 2024-03-25T18:28:17.857 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 10 monclient: init 2024-03-25T18:28:17.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:17.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 10 monclient(hunting): picked mon.noname-a con 0x7fc2540b4700 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:17.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 10 monclient(hunting): picked mon.noname-b con 0x7fc254142a20 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:17.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 10 monclient(hunting): picked mon.noname-c con 0x7fc2540a96b0 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:17.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2540a9d60 con 0x7fc2540b4700 2024-03-25T18:28:17.859 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25413ff10 con 0x7fc2540a96b0 2024-03-25T18:28:17.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.855+0000 7fc25923e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2540aa1f0 con 0x7fc254142a20 2024-03-25T18:28:17.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.856+0000 7fc25923e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.856+0000 7fc25923e640 10 monclient(hunting): authenticate will time out at 7302.113377s 2024-03-25T18:28:17.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.856+0000 7fc252d9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fc2540b4700 0x7fc254142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:50790/0 (socket says 172.21.15.55:50790) 2024-03-25T18:28:17.860 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.856+0000 7fc252d9d640 1 -- 172.21.15.55:0/316767166 learned_addr learned my addr 172.21.15.55:0/316767166 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697300742 0 0) 0x7fc2540aa1f0 con 0x7fc254142a20 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc23c003520 con 0x7fc254142a20 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247856281 0 0) 0x7fc2540a9d60 con 0x7fc2540b4700 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.862 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.859+0000 7fc251d9b640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:28:17.863 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.860+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2540aa1f0 con 0x7fc2540b4700 2024-03-25T18:28:17.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1533527064 0 0) 0x7fc23c003520 con 0x7fc254142a20 2024-03-25T18:28:17.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 >> v1:172.21.15.133:6789/0 conn(0x7fc2540a96b0 legacy=0x7fc25413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 >> v1:172.21.15.55:6789/0 conn(0x7fc2540b4700 legacy=0x7fc254142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.865 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 1 monclient: found mon.noname-b 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2540b4ae0 con 0x7fc254142a20 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fc248001690 con 0x7fc254142a20 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.863+0000 7fc251d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.864+0000 7fc251d9b640 10 monclient: dump: 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.866 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.867 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.867 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.867 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:17.867 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.867 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.867 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.867 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:17.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.864+0000 7fc25923e640 5 monclient: authenticate success, global_id 4166 2024-03-25T18:28:17.867 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.864+0000 7fc25923e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:17.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.890+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2480017f0 con 0x7fc254142a20 2024-03-25T18:28:17.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc251d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:17.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc251d9b640 1 -- 172.21.15.55:0/316767166 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fc248001c00 con 0x7fc254142a20 2024-03-25T18:28:17.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc251d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc251d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:17.893 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc251d9b640 10 monclient: dump: 2024-03-25T18:28:17.893 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-25T18:28:17.893 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.893 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc25923e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc25923e640 4 set_mon_vals no callback set 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc25923e640 10 monclient: shutdown 2024-03-25T18:28:17.894 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc25923e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:17.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc25923e640 1 -- 172.21.15.55:0/316767166 >> v1:172.21.15.55:6790/0 conn(0x7fc254142a20 legacy=0x7fc25413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc25923e640 1 -- 172.21.15.55:0/316767166 shutdown_connections 2024-03-25T18:28:17.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.891+0000 7fc25923e640 1 -- 172.21.15.55:0/316767166 >> 172.21.15.55:0/316767166 conn(0x7fc2540668e0 msgr2=0x7fc254066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:17.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.892+0000 7fc25923e640 1 -- 172.21.15.55:0/316767166 shutdown_connections 2024-03-25T18:28:17.895 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.892+0000 7fc25923e640 1 -- 172.21.15.55:0/316767166 wait complete. 2024-03-25T18:28:17.898 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.893+0000 7fc25923e640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.893+0000 7fc25923e640 10 monclient: monmap: 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 0 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 0.000000 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:created 0.000000 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 1 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.893+0000 7fc25923e640 1 librados: starting msgr at 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.894+0000 7fc25923e640 1 librados: starting objecter 2024-03-25T18:28:17.899 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.894+0000 7fc25923e640 1 Processor -- start 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.894+0000 7fc25923e640 1 -- start start 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.894+0000 7fc25923e640 1 librados: setting wanted keys 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.894+0000 7fc25923e640 1 librados: calling monclient init 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.894+0000 7fc25923e640 10 monclient: init 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.895+0000 7fc25923e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.895+0000 7fc25923e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.895+0000 7fc25923e640 10 monclient(hunting): picked mon.noname-c con 0x7fc2540a96b0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.895+0000 7fc25923e640 10 monclient(hunting): picked mon.noname-b con 0x7fc2540b4700 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.895+0000 7fc25923e640 10 monclient(hunting): picked mon.noname-a con 0x7fc254142a20 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.896+0000 7fc25923e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2540ac100 con 0x7fc254142a20 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.896+0000 7fc25923e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2540ac3e0 con 0x7fc2540b4700 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.896+0000 7fc25923e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2540ac6c0 con 0x7fc2540a96b0 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.896+0000 7fc25923e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.900 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.896+0000 7fc25923e640 10 monclient(hunting): authenticate will time out at 7302.153704s 2024-03-25T18:28:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.896+0000 7fc25259c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fc2540b4700 0x7fc25413c490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:49786/0 (socket says 172.21.15.55:49786) 2024-03-25T18:28:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.896+0000 7fc25259c640 1 -- 172.21.15.55:0/1606770823 learned_addr learned my addr 172.21.15.55:0/1606770823 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.897+0000 7fc23b7fe640 10 client.?.objecter ms_handle_connect 0x7fc254142a20 2024-03-25T18:28:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.897+0000 7fc23b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.897+0000 7fc23b7fe640 10 client.?.objecter ms_handle_connect 0x7fc2540a96b0 2024-03-25T18:28:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.897+0000 7fc23b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.897+0000 7fc23b7fe640 10 client.?.objecter ms_handle_connect 0x7fc2540b4700 2024-03-25T18:28:17.901 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.897+0000 7fc23b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.899+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86887843 0 0) 0x7fc2540ac6c0 con 0x7fc2540a96b0 2024-03-25T18:28:17.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.899+0000 7fc23b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.899+0000 7fc23b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.899+0000 7fc23b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.899+0000 7fc23b7fe640 10 monclient(hunting): my global_id is 4169 2024-03-25T18:28:17.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.899+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc240004eb0 con 0x7fc2540a96b0 2024-03-25T18:28:17.902 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941442753 0 0) 0x7fc2540ac100 con 0x7fc254142a20 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2540ac6c0 con 0x7fc254142a20 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278250366 0 0) 0x7fc2540ac3e0 con 0x7fc2540b4700 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 10 monclient(hunting): my global_id is 4168 2024-03-25T18:28:17.903 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.900+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2540ac100 con 0x7fc2540b4700 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.904+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4200422315 0 0) 0x7fc240004eb0 con 0x7fc2540a96b0 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.904+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2540ac3e0 con 0x7fc2540a96b0 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.905+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fc244001930 con 0x7fc2540a96b0 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.905+0000 7fc23b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.905+0000 7fc23b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.905+0000 7fc23b7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.906 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:17.907 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.907 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:17.907 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:17.907 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:17.907 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:17.907 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:17.907 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.905+0000 7fc23b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:17.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.907+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 628327339 0 0) 0x7fc2540ac100 con 0x7fc2540b4700 2024-03-25T18:28:17.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.907+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc240004eb0 con 0x7fc2540b4700 2024-03-25T18:28:17.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.908+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 306432026 0 0) 0x7fc2540ac6c0 con 0x7fc254142a20 2024-03-25T18:28:17.909 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.908+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2540ac100 con 0x7fc254142a20 2024-03-25T18:28:17.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2646245474 0 0) 0x7fc2540ac3e0 con 0x7fc2540a96b0 2024-03-25T18:28:17.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 >> v1:172.21.15.133:6789/0 conn(0x7fc2540b4700 legacy=0x7fc25413c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 >> v1:172.21.15.55:6789/0 conn(0x7fc254142a20 legacy=0x7fc2540acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.911 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 1 monclient: found mon.c 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2540ac9a0 con 0x7fc2540a96b0 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc23b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc25923e640 5 monclient: authenticate success, global_id 4169 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc25923e640 10 monclient: _renew_subs 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc25923e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.909+0000 7fc25923e640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25413b880 con 0x7fc2540a96b0 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.910+0000 7fc25923e640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.910+0000 7fc25923e640 10 monclient: _renew_subs 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.910+0000 7fc25923e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:28:17.912 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.910+0000 7fc25923e640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2541e3690 con 0x7fc2540a96b0 2024-03-25T18:28:17.913 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.910+0000 7fc25923e640 1 librados: init done 2024-03-25T18:28:17.913 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:17.912+0000 7f6a26412040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:28:17.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.911+0000 7fc25923e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.911+0000 7fc25923e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.911+0000 7fc25923e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:28:17.914 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:17.911+0000 7fc25923e640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25413cc60 con 0x7fc2540a96b0 2024-03-25T18:28:18.075 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:18.074+0000 7fa1e3641040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:28:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2440047e0 con 0x7fc2540a96b0 2024-03-25T18:28:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fc244004af0 con 0x7fc2540a96b0 2024-03-25T18:28:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 10 monclient: dump: 2024-03-25T18:28:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:epoch 1 2024-03-25T18:28:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc2440050a0 con 0x7fc2540a96b0 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fc244006970 con 0x7fc2540a96b0 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 10 client.4169.objecter ms_dispatch 0x7fc254000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 3 client.4169.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 3 client.4169.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 20 client.4169.objecter dump_active .. 0 homeless 2024-03-25T18:28:18.121 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.118+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc244018de0 con 0x7fc2540a96b0 2024-03-25T18:28:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.119+0000 7fc23b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.119+0000 7fc23b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.119+0000 7fc250d99640 4 set_mon_vals no callback set 2024-03-25T18:28:18.165 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:18.164+0000 7fa1e3641040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:28:18.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.187+0000 7fc25923e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:18.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.187+0000 7fc25923e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:18.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.187+0000 7fc25923e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:28:18.188 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.187+0000 7fc25923e640 1 -- 172.21.15.55:0/1606770823 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc25404f730 con 0x7fc2540a96b0 2024-03-25T18:28:18.213 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:18.212+0000 7f6a26412040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:28:18.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.257+0000 7fc23b7fe640 1 -- 172.21.15.55:0/1606770823 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fc2440062a0 con 0x7fc2540a96b0 2024-03-25T18:28:18.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.257+0000 7fc23b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:18.258 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.257+0000 7fc23b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:18.259 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-03-25T18:28:18.259 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:18.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.258+0000 7fc25923e640 10 librados: watch_flush enter 2024-03-25T18:28:18.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.258+0000 7fc25923e640 10 librados: watch_flush exit 2024-03-25T18:28:18.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.258+0000 7fc25923e640 20 client.4169.objecter shutdown clearing up homeless session... 2024-03-25T18:28:18.260 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.258+0000 7fc25923e640 10 client.4169.objecter successfully canceled tick 2024-03-25T18:28:18.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.259+0000 7fc25923e640 10 monclient: shutdown 2024-03-25T18:28:18.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.259+0000 7fc25923e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:18.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.259+0000 7fc25923e640 1 -- 172.21.15.55:0/1606770823 >> v1:172.21.15.55:6790/0 conn(0x7fc2540a96b0 legacy=0x7fc25413a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.259+0000 7fc25923e640 1 -- 172.21.15.55:0/1606770823 shutdown_connections 2024-03-25T18:28:18.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.259+0000 7fc25923e640 1 -- 172.21.15.55:0/1606770823 >> 172.21.15.55:0/1606770823 conn(0x7fc2540668e0 msgr2=0x7fc25413e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:18.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.259+0000 7fc25923e640 1 -- 172.21.15.55:0/1606770823 shutdown_connections 2024-03-25T18:28:18.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.259+0000 7fc25923e640 1 -- 172.21.15.55:0/1606770823 wait complete. 2024-03-25T18:28:18.261 INFO:teuthology.orchestra.run.smithi055.stderr:2024-03-25T18:28:18.259+0000 7fc25923e640 1 librados: shutdown 2024-03-25T18:28:18.269 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:18.268+0000 7f6a26412040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:28:18.274 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:28:18.275 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:28:18.381 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:18.380+0000 7fa1e3641040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:28:18.394 INFO:tasks.ceph.mgr.x.smithi133.stderr:2024-03-25T18:28:18.393+0000 7f6a26412040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:28:19.082 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.227+0000 7f8f82810640 10 monclient: get_monmap_and_config 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.227+0000 7f8f82810640 10 monclient: build_initial_monmap 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.227+0000 7f8f82810640 10 monclient: monmap: 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:28:19.227900+0000 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:28:19.227900+0000 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:19.228 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:19.229 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:19.229 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:19.229 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:19.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.228+0000 7f8f82810640 1 Processor -- start 2024-03-25T18:28:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.228+0000 7f8f82810640 1 -- start start 2024-03-25T18:28:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.228+0000 7f8f82810640 10 monclient: init 2024-03-25T18:28:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:19.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 10 monclient(hunting): picked mon.noname-b con 0x7f8f740389e0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 10 monclient(hunting): picked mon.noname-c con 0x7f8f74038dc0 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 10 monclient(hunting): picked mon.noname-a con 0x7f8f74030660 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f740c54c0 con 0x7f8f74030660 2024-03-25T18:28:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f74036430 con 0x7f8f74038dc0 2024-03-25T18:28:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f74039840 con 0x7f8f740389e0 2024-03-25T18:28:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:19.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.229+0000 7f8f82810640 10 monclient(hunting): authenticate will time out at 7303.487180s 2024-03-25T18:28:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.235+0000 7f8f8180e640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f8f740389e0 0x7f8f74033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:48596/0 (socket says 172.21.15.55:48596) 2024-03-25T18:28:19.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.235+0000 7f8f8180e640 1 -- 172.21.15.55:0/2687480071 learned_addr learned my addr 172.21.15.55:0/2687480071 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:19.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.247+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137348877 0 0) 0x7f8f74039840 con 0x7f8f740389e0 2024-03-25T18:28:19.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.247+0000 7f8f8080c640 10 monclient(hunting): _negotiate 2024-03-25T18:28:19.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.247+0000 7f8f8080c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:19.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.247+0000 7f8f8080c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:19.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.247+0000 7f8f8080c640 10 monclient(hunting): my global_id is 4181 2024-03-25T18:28:19.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.247+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f68003520 con 0x7f8f740389e0 2024-03-25T18:28:19.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.254+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157256956 0 0) 0x7f8f740c54c0 con 0x7f8f74030660 2024-03-25T18:28:19.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.254+0000 7f8f8080c640 10 monclient(hunting): _negotiate 2024-03-25T18:28:19.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.254+0000 7f8f8080c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:19.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.254+0000 7f8f8080c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:19.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.254+0000 7f8f8080c640 10 monclient(hunting): my global_id is 4182 2024-03-25T18:28:19.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.254+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f74039840 con 0x7f8f74030660 2024-03-25T18:28:19.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4162406499 0 0) 0x7f8f68003520 con 0x7f8f740389e0 2024-03-25T18:28:19.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 >> v1:172.21.15.133:6789/0 conn(0x7f8f74038dc0 legacy=0x7f8f740cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:28:19.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 >> v1:172.21.15.55:6789/0 conn(0x7f8f74030660 legacy=0x7f8f740d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:19.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 10 monclient: _finish_hunting 0 2024-03-25T18:28:19.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 1 monclient: found mon.noname-b 2024-03-25T18:28:19.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:19.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:28:19.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f74035690 con 0x7f8f740389e0 2024-03-25T18:28:19.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 10 monclient: _finish_auth 0 2024-03-25T18:28:19.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 10 monclient: _check_auth_tickets 2024-03-25T18:28:19.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:19.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8f6c001690 con 0x7f8f740389e0 2024-03-25T18:28:19.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:19.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:19.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f8080c640 10 monclient: dump: 2024-03-25T18:28:19.261 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:19.261 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:19.261 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:19.262 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:19.262 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:19.262 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:19.262 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:19.262 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:19.262 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:19.262 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:19.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f82810640 5 monclient: authenticate success, global_id 4181 2024-03-25T18:28:19.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.258+0000 7f8f82810640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:19.272 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.272+0000 7f8f8080c640 1 -- 172.21.15.55:0/2687480071 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f6c001920 con 0x7f8f740389e0 2024-03-25T18:28:19.272 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.272+0000 7f8f8080c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:19.272 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.272+0000 7f8f82810640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:19.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.272+0000 7f8f82810640 4 set_mon_vals no callback set 2024-03-25T18:28:19.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.273+0000 7f8f82810640 10 monclient: shutdown 2024-03-25T18:28:19.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.273+0000 7f8f82810640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:19.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.273+0000 7f8f82810640 1 -- 172.21.15.55:0/2687480071 >> v1:172.21.15.55:6790/0 conn(0x7f8f740389e0 legacy=0x7f8f74033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:19.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.273+0000 7f8f82810640 1 -- 172.21.15.55:0/2687480071 shutdown_connections 2024-03-25T18:28:19.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.273+0000 7f8f82810640 1 -- 172.21.15.55:0/2687480071 >> 172.21.15.55:0/2687480071 conn(0x7f8f74005c00 msgr2=0x7f8f74006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:19.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.273+0000 7f8f82810640 1 -- 172.21.15.55:0/2687480071 shutdown_connections 2024-03-25T18:28:19.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.273+0000 7f8f82810640 1 -- 172.21.15.55:0/2687480071 wait complete. 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.274+0000 7f8f82810640 10 monclient: build_initial_monmap 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.274+0000 7f8f82810640 10 monclient: monmap: 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.274+0000 7f8f82810640 1 librados: starting msgr at 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.274+0000 7f8f82810640 1 librados: starting objecter 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.275+0000 7f8f82810640 1 Processor -- start 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.275+0000 7f8f82810640 1 -- start start 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.275+0000 7f8f82810640 1 librados: setting wanted keys 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.275+0000 7f8f82810640 1 librados: calling monclient init 2024-03-25T18:28:19.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.275+0000 7f8f82810640 10 monclient: init 2024-03-25T18:28:19.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:19.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:19.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 10 monclient(hunting): picked mon.noname-b con 0x7f8f74030660 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:19.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 10 monclient(hunting): picked mon.noname-a con 0x7f8f740389e0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:19.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 10 monclient(hunting): picked mon.noname-c con 0x7f8f740c80d0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:19.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f74009770 con 0x7f8f740389e0 2024-03-25T18:28:19.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f74009900 con 0x7f8f74030660 2024-03-25T18:28:19.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f74009a90 con 0x7f8f740c80d0 2024-03-25T18:28:19.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:19.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.276+0000 7f8f82810640 10 monclient(hunting): authenticate will time out at 7303.534123s 2024-03-25T18:28:19.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.281+0000 7f8f8200f640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f8f740c80d0 0x7f8f7416b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:48624/0 (socket says 172.21.15.55:48624) 2024-03-25T18:28:19.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.281+0000 7f8f8200f640 1 -- 172.21.15.55:0/3069774477 learned_addr learned my addr 172.21.15.55:0/3069774477 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:19.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.282+0000 7f8f727fc640 10 client.?.objecter ms_handle_connect 0x7f8f740c80d0 2024-03-25T18:28:19.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.282+0000 7f8f727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:19.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.289+0000 7f8f727fc640 10 client.?.objecter ms_handle_connect 0x7f8f740389e0 2024-03-25T18:28:19.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.289+0000 7f8f727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:19.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.297+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662079362 0 0) 0x7f8f74009770 con 0x7f8f740389e0 2024-03-25T18:28:19.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.297+0000 7f8f727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:19.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.297+0000 7f8f727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:19.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.297+0000 7f8f727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:19.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.297+0000 7f8f727fc640 10 monclient(hunting): my global_id is 4188 2024-03-25T18:28:19.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.297+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f60004eb0 con 0x7f8f740389e0 2024-03-25T18:28:19.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.333+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2915788359 0 0) 0x7f8f60004eb0 con 0x7f8f740389e0 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.334+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f74009770 con 0x7f8f740389e0 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.334+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8f780017d0 con 0x7f8f740389e0 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.334+0000 7f8f727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.334+0000 7f8f727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.334+0000 7f8f727fc640 10 monclient(hunting): dump: 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:19.335 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:19.336 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:19.336 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:19.336 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:19.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.334+0000 7f8f727fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 956492754 0 0) 0x7f8f74009770 con 0x7f8f740389e0 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 >> v1:172.21.15.55:6790/0 conn(0x7f8f740c80d0 legacy=0x7f8f7416b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 >> v1:172.21.15.133:6789/0 conn(0x7f8f74030660 legacy=0x7f8f74160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 1 monclient: found mon.a 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f741647c0 con 0x7f8f740389e0 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:19.341 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f82810640 5 monclient: authenticate success, global_id 4188 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f82810640 10 monclient: _renew_subs 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f82810640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.339+0000 7f8f82810640 1 -- 172.21.15.55:0/3069774477 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f74164950 con 0x7f8f740389e0 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.340+0000 7f8f82810640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.340+0000 7f8f82810640 10 monclient: _renew_subs 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.340+0000 7f8f82810640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.340+0000 7f8f82810640 1 -- 172.21.15.55:0/3069774477 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f74164ca0 con 0x7f8f740389e0 2024-03-25T18:28:19.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.340+0000 7f8f82810640 1 librados: init done 2024-03-25T18:28:19.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.341+0000 7f8f82810640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:19.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.341+0000 7f8f82810640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:19.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.341+0000 7f8f82810640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:19.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.341+0000 7f8f82810640 1 -- 172.21.15.55:0/3069774477 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7400a9b0 con 0x7f8f740389e0 2024-03-25T18:28:19.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.563+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f78001930 con 0x7f8f740389e0 2024-03-25T18:28:19.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.563+0000 7f8f727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:19.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.563+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8f78003790 con 0x7f8f740389e0 2024-03-25T18:28:19.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.563+0000 7f8f727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:19.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.563+0000 7f8f727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:19.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.563+0000 7f8f727fc640 10 monclient: dump: 2024-03-25T18:28:19.564 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.563+0000 7f8f73fff640 4 set_mon_vals no callback set 2024-03-25T18:28:19.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.563+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2459588574 0 0) 0x7f8f7800e930 con 0x7f8f740389e0 2024-03-25T18:28:19.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.616+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f8f78014480 con 0x7f8f740389e0 2024-03-25T18:28:19.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.616+0000 7f8f727fc640 10 client.4188.objecter ms_dispatch 0x7f8f7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:19.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.616+0000 7f8f727fc640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:19.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.616+0000 7f8f727fc640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:19.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.616+0000 7f8f727fc640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-25T18:28:19.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.616+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8f78013740 con 0x7f8f740389e0 2024-03-25T18:28:19.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.616+0000 7f8f727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:19.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.616+0000 7f8f727fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:19.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.745+0000 7f8f82810640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:19.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.745+0000 7f8f82810640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:19.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.745+0000 7f8f82810640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:19.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.745+0000 7f8f82810640 1 -- 172.21.15.55:0/3069774477 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f74007b90 con 0x7f8f740389e0 2024-03-25T18:28:19.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.881+0000 7f8f727fc640 1 -- 172.21.15.55:0/3069774477 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f8f78011f20 con 0x7f8f740389e0 2024-03-25T18:28:19.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.881+0000 7f8f727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:19.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.881+0000 7f8f727fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:19.883 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:28:19.883 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:19.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.882+0000 7f8f82810640 10 librados: watch_flush enter 2024-03-25T18:28:19.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.882+0000 7f8f82810640 10 librados: watch_flush exit 2024-03-25T18:28:19.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.882+0000 7f8f82810640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-25T18:28:19.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.882+0000 7f8f82810640 10 client.4188.objecter successfully canceled tick 2024-03-25T18:28:19.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.882+0000 7f8f82810640 10 monclient: shutdown 2024-03-25T18:28:19.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.882+0000 7f8f82810640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:19.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.882+0000 7f8f82810640 1 -- 172.21.15.55:0/3069774477 >> v1:172.21.15.55:6789/0 conn(0x7f8f740389e0 legacy=0x7f8f74161520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:19.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.883+0000 7f8f82810640 1 -- 172.21.15.55:0/3069774477 shutdown_connections 2024-03-25T18:28:19.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.883+0000 7f8f82810640 1 -- 172.21.15.55:0/3069774477 >> 172.21.15.55:0/3069774477 conn(0x7f8f74005c00 msgr2=0x7f8f74034b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:19.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.883+0000 7f8f82810640 1 -- 172.21.15.55:0/3069774477 shutdown_connections 2024-03-25T18:28:19.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.883+0000 7f8f82810640 1 -- 172.21.15.55:0/3069774477 wait complete. 2024-03-25T18:28:19.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:19.883+0000 7f8f82810640 1 librados: shutdown 2024-03-25T18:28:19.897 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:21.600 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:21.600+0000 7fa1842d1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:28:21.670 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:28:21.669+0000 7fa1842d1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:28:24.503 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:06.716 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:24.586 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:06.798 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:24.739 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:06.942 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:24.798 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:07.002 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:24.848 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:07.057 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:24.907 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:07.116 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.042 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:07.225 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.105 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:07.288 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.131 INFO:tasks.ceph.osd.4.smithi133.stderr:2024-03-25T18:28:25.125+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:28:25.318 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:07.513 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.374 INFO:tasks.ceph.osd.6.smithi133.stderr:2024-03-25T18:28:25.367+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:28:25.379 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:07.574 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.392 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:07.587 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.395 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:07.596 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.454 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:07.648 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.458 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:07.659 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.464 INFO:tasks.ceph.osd.5.smithi133.stderr:2024-03-25T18:28:25.458+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:28:25.493 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:07.713 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.553 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:07.773 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.633 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:07.845 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.676 INFO:tasks.ceph.osd.7.smithi133.stderr:2024-03-25T18:28:25.667+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:28:25.784 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:07.996 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.846 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:08.049 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.920 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:08.128 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:25.932 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-25T18:28:25.923+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:28:26.010 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-25T18:28:26.003+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:28:26.016 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-25T18:28:26.010+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:28:26.105 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:08.317 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.121 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-25T18:28:26.114+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:28:26.139 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:08.323 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.233 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:08.437 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.296 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:08.505 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.370 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:08.582 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.407 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:08.602 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.486 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:08.686 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.487 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:08.682 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.521 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:08.705 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.551 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:08.754 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.595 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:08.815 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.643 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:08.855 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.654 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:08.863 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.703 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:26.775 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:08.970 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.791+0000 7f97a2dc7640 10 monclient: get_monmap_and_config 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.791+0000 7f97a2dc7640 10 monclient: build_initial_monmap 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.791+0000 7f97a2dc7640 10 monclient: monmap: 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:28:26.792317+0000 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:28:26.792317+0000 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:26.792 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:26.793 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:26.793 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:26.793 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 1 Processor -- start 2024-03-25T18:28:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 1 -- start start 2024-03-25T18:28:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 10 monclient: init 2024-03-25T18:28:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 10 monclient(hunting): picked mon.noname-c con 0x7f979c0b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 10 monclient(hunting): picked mon.noname-a con 0x7f979c0b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 10 monclient(hunting): picked mon.noname-b con 0x7f979c14be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c0af940 con 0x7f979c0b1730 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c0b0a70 con 0x7f979c0b4d60 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c0b0d50 con 0x7f979c14be90 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.792+0000 7f97a2dc7640 10 monclient(hunting): authenticate will time out at 7311.050275s 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.793+0000 7f97a1364640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f979c14be90 0x7f979c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:48672/0 (socket says 172.21.15.55:48672) 2024-03-25T18:28:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.793+0000 7f97a1364640 1 -- 172.21.15.55:0/3909748184 learned_addr learned my addr 172.21.15.55:0/3909748184 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.795+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359717087 0 0) 0x7f979c0b0a70 con 0x7f979c0b4d60 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.795+0000 7f979b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.795+0000 7f979b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.795+0000 7f979b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.795+0000 7f979b7fe640 10 monclient(hunting): my global_id is 4198 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.795+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f978c003520 con 0x7f979c0b4d60 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.796+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895811439 0 0) 0x7f979c0b0d50 con 0x7f979c14be90 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.796+0000 7f979b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.796+0000 7f979b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.796+0000 7f979b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.796+0000 7f979b7fe640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:28:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.796+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c0b0a70 con 0x7f979c14be90 2024-03-25T18:28:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.797+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077375351 0 0) 0x7f979c0af940 con 0x7f979c0b1730 2024-03-25T18:28:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.797+0000 7f979b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.797+0000 7f979b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.797+0000 7f979b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.797+0000 7f979b7fe640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:28:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.797+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c0b0d50 con 0x7f979c0b1730 2024-03-25T18:28:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.798+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1700076052 0 0) 0x7f978c003520 con 0x7f979c0b4d60 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.798+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 >> v1:172.21.15.55:6790/0 conn(0x7f979c14be90 legacy=0x7f979c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.798+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 >> v1:172.21.15.55:6789/0 conn(0x7f979c0b1730 legacy=0x7f979c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c0b1030 con 0x7f979c0b4d60 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9790001690 con 0x7f979c0b4d60 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f979b7fe640 10 monclient: dump: 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:26.801 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:26.802 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:26.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f97a2dc7640 5 monclient: authenticate success, global_id 4198 2024-03-25T18:28:26.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.799+0000 7f97a2dc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:26.803 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.800+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97900017f0 con 0x7f979c0b4d60 2024-03-25T18:28:26.803 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.800+0000 7f979b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:26.803 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.800+0000 7f97a2dc7640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.800+0000 7f97a2dc7640 4 set_mon_vals no callback set 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.800+0000 7f979b7fe640 1 -- 172.21.15.55:0/3909748184 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9790001c00 con 0x7f979c0b4d60 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.800+0000 7f979b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.800+0000 7f979b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.800+0000 7f979b7fe640 10 monclient: dump: 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:26.804 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.801+0000 7f97a2dc7640 10 monclient: shutdown 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.801+0000 7f97a2dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.801+0000 7f97a2dc7640 1 -- 172.21.15.55:0/3909748184 >> v1:172.21.15.133:6789/0 conn(0x7f979c0b4d60 legacy=0x7f979c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.801+0000 7f97a2dc7640 1 -- 172.21.15.55:0/3909748184 shutdown_connections 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.801+0000 7f97a2dc7640 1 -- 172.21.15.55:0/3909748184 >> 172.21.15.55:0/3909748184 conn(0x7f979c0617a0 msgr2=0x7f979c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.801+0000 7f97a2dc7640 1 -- 172.21.15.55:0/3909748184 shutdown_connections 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.801+0000 7f97a2dc7640 1 -- 172.21.15.55:0/3909748184 wait complete. 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.802+0000 7f97a2dc7640 10 monclient: build_initial_monmap 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.802+0000 7f97a2dc7640 10 monclient: monmap: 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:26.805 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.802+0000 7f97a2dc7640 1 librados: starting msgr at 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.802+0000 7f97a2dc7640 1 librados: starting objecter 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.802+0000 7f97a2dc7640 1 Processor -- start 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.803+0000 7f97a2dc7640 1 -- start start 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.803+0000 7f97a2dc7640 1 librados: setting wanted keys 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.803+0000 7f97a2dc7640 1 librados: calling monclient init 2024-03-25T18:28:26.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.803+0000 7f97a2dc7640 10 monclient: init 2024-03-25T18:28:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.803+0000 7f97a2dc7640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.803+0000 7f97a2dc7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.803+0000 7f97a2dc7640 10 monclient(hunting): picked mon.noname-a con 0x7f979c0b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97a2dc7640 10 monclient(hunting): picked mon.noname-b con 0x7f979c0b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97a2dc7640 10 monclient(hunting): picked mon.noname-c con 0x7f979c14be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97a2dc7640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c141ca0 con 0x7f979c0b1730 2024-03-25T18:28:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97a2dc7640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c1d7940 con 0x7f979c0b4d60 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97a2dc7640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c140d00 con 0x7f979c14be90 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97a2dc7640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97a2dc7640 10 monclient(hunting): authenticate will time out at 7311.061466s 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f979bfff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f979c0b4d60 0x7f979c1d7e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:42346/0 (socket says 172.21.15.55:42346) 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f979bfff640 1 -- 172.21.15.55:0/2588796241 learned_addr learned my addr 172.21.15.55:0/2588796241 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97a1364640 1 --1- 172.21.15.55:0/2588796241 >> v1:172.21.15.55:6790/0 conn(0x7f979c14be90 0x7f979c142c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:48680/0 (socket says 172.21.15.55:48680) 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97997fa640 10 client.?.objecter ms_handle_connect 0x7f979c0b4d60 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.804+0000 7f97997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.805+0000 7f97997fa640 10 client.?.objecter ms_handle_connect 0x7f979c14be90 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.805+0000 7f97997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.805+0000 7f97997fa640 10 client.?.objecter ms_handle_connect 0x7f979c0b1730 2024-03-25T18:28:26.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.805+0000 7f97997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.806+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653870494 0 0) 0x7f979c1d7940 con 0x7f979c0b4d60 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.806+0000 7f97997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.806+0000 7f97997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.806+0000 7f97997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.806+0000 7f97997fa640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.806+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9780004eb0 con 0x7f979c0b4d60 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411326406 0 0) 0x7f979c140d00 con 0x7f979c14be90 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c1d7940 con 0x7f979c14be90 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893855714 0 0) 0x7f979c141ca0 con 0x7f979c0b1730 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:28:26.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.807+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c140d00 con 0x7f979c0b1730 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63618654 0 0) 0x7f9780004eb0 con 0x7f979c0b4d60 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f979c141ca0 con 0x7f979c0b4d60 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9788001a10 con 0x7f979c0b4d60 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 10 monclient(hunting): dump: 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.813 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1957941472 0 0) 0x7f979c1d7940 con 0x7f979c14be90 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9780004eb0 con 0x7f979c14be90 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9794001b50 con 0x7f979c14be90 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.811+0000 7f97997fa640 10 monclient(hunting): dump: 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:26.814 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:26.815 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.815 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.815 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.815 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:26.815 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:26.815 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:26.815 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:26.815 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:26.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1420415480 0 0) 0x7f979c141ca0 con 0x7f979c0b4d60 2024-03-25T18:28:26.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 >> v1:172.21.15.55:6790/0 conn(0x7f979c14be90 legacy=0x7f979c142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 >> v1:172.21.15.55:6789/0 conn(0x7f979c0b1730 legacy=0x7f979c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:26.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 1 monclient: found mon.b 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c1d8570 con 0x7f979c0b4d60 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97a2dc7640 5 monclient: authenticate success, global_id 4201 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97a2dc7640 10 monclient: _renew_subs 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97a2dc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c13b370 con 0x7f979c0b4d60 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97a2dc7640 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97a2dc7640 10 monclient: _renew_subs 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97a2dc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c143550 con 0x7f979c0b4d60 2024-03-25T18:28:26.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.814+0000 7f97a2dc7640 1 librados: init done 2024-03-25T18:28:26.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.815+0000 7f97a2dc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:26.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.815+0000 7f97a2dc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:26.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.815+0000 7f97a2dc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:26.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.815+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979c1d77a0 con 0x7f979c0b4d60 2024-03-25T18:28:26.818 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:09.020 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.835+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9788001c70 con 0x7f979c0b4d60 2024-03-25T18:28:26.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.835+0000 7f97997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.835+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f97880038c0 con 0x7f979c0b4d60 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.835+0000 7f97997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.835+0000 7f97997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.835+0000 7f97997fa640 10 monclient: dump: 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:26.836 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:26.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.835+0000 7f979affd640 4 set_mon_vals no callback set 2024-03-25T18:28:26.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.835+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f9788003ae0 con 0x7f979c0b4d60 2024-03-25T18:28:26.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.836+0000 7f97a0b63640 10 monclient: get_auth_request con 0x7f978006dd70 auth_method 0 2024-03-25T18:28:26.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.836+0000 7f97997fa640 10 client.4201.objecter ms_handle_connect 0x7f978006dd70 2024-03-25T18:28:26.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.837+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f9788004460 con 0x7f979c0b4d60 2024-03-25T18:28:26.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.837+0000 7f97997fa640 10 client.4201.objecter ms_dispatch 0x7f979c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:26.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.837+0000 7f97997fa640 3 client.4201.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:26.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.837+0000 7f97997fa640 3 client.4201.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:26.838 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:26.837+0000 7f97997fa640 20 client.4201.objecter dump_active .. 0 homeless 2024-03-25T18:28:26.868 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:09.063 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.884 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:09.067 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.884 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:09.085 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.915 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:09.127 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.942 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:09.151 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:26.969 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:09.189 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.068+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97880501c0 con 0x7f979c0b4d60 2024-03-25T18:28:27.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.068+0000 7f97997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:27.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.068+0000 7f97997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:27.084 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:09.288 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.128 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:09.323 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.172 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:09.356 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.182 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:09.377 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.191 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:09.404 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.231 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:09.440 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.248 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:09.448 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.255+0000 7f97a2dc7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:27.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.255+0000 7f97a2dc7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:27.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.255+0000 7f97a2dc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:28:27.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.255+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f979c1d8700 con 0x7f979c0b4d60 2024-03-25T18:28:27.273 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:09.482 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.281 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:09.501 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.303+0000 7f97997fa640 1 -- 172.21.15.55:0/2588796241 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f97880520f0 con 0x7f979c0b4d60 2024-03-25T18:28:27.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.303+0000 7f97997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:27.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.303+0000 7f97997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:27.305 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:28:27.305 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:27.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.305+0000 7f97a2dc7640 10 librados: watch_flush enter 2024-03-25T18:28:27.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 10 librados: watch_flush exit 2024-03-25T18:28:27.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 20 client.4201.objecter shutdown clearing up homeless session... 2024-03-25T18:28:27.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 10 client.4201.objecter successfully canceled tick 2024-03-25T18:28:27.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 >> v1:172.21.15.55:6800/286536064 conn(0x7f978006dd70 legacy=0x7f9780070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:27.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 10 monclient: shutdown 2024-03-25T18:28:27.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:27.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 >> v1:172.21.15.133:6789/0 conn(0x7f979c0b4d60 legacy=0x7f979c1d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:27.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 shutdown_connections 2024-03-25T18:28:27.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 >> 172.21.15.55:0/2588796241 conn(0x7f979c0617a0 msgr2=0x7f979c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:27.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 shutdown_connections 2024-03-25T18:28:27.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 1 -- 172.21.15.55:0/2588796241 wait complete. 2024-03-25T18:28:27.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:27.306+0000 7f97a2dc7640 1 librados: shutdown 2024-03-25T18:28:27.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:27.349 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:09.552 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.421 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:09.616 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.452 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:09.647 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.463 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:09.675 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.468 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:09.651 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.545 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:09.745 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.549 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:09.769 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.561 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:09.770 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.617 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:09.820 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.709 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:09.904 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.725 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:09.920 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.726 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:09.939 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.756 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:09.940 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.817 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:10.037 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.840 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:10.041 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.848 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:10.057 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:27.892 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:10.095 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.004 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:10.199 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.010 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:10.206 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.051 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:10.235 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.085 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:10.305 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.131 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:10.331 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.135 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:10.344 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.158 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:10.362 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.193 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:10.405 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.245 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:10.457 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.268 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:10.463 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.301 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:10.495 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.344 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:10.528 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.350 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:10.570 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.407 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:10.615 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.420 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:10.620 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.461 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:10.669 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.542 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:10.735 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.590 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:10.785 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.621 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:10.841 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.633 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:10.836 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.636 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:10.820 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.710 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:10.910 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.711 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:10.915 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.804 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:10.999 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.881 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:11.076 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.895 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:11.115 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:28.999 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:11.199 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.247 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:11.431 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.281 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:11.475 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.310 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:11.494 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.361 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:11.555 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.381 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:11.602 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.437 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:11.657 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.449 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:11.644 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.510 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:11.705 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.539 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:11.739 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:29.623 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:11.823 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:33.836 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:16.048 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:34.125 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:34.176 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:16.379 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:34.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.216+0000 7ff2d3cf8640 10 monclient: get_monmap_and_config 2024-03-25T18:28:34.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.216+0000 7ff2d3cf8640 10 monclient: build_initial_monmap 2024-03-25T18:28:34.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.216+0000 7ff2d3cf8640 10 monclient: monmap: 2024-03-25T18:28:34.218 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:34.218 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:34.218 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:28:34.218404+0000 2024-03-25T18:28:34.219 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:28:34.218404+0000 2024-03-25T18:28:34.219 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:34.219 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:34.219 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:34.219 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:34.219 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:34.219 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:34.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.217+0000 7ff2d3cf8640 1 Processor -- start 2024-03-25T18:28:34.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.217+0000 7ff2d3cf8640 1 -- start start 2024-03-25T18:28:34.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.217+0000 7ff2d3cf8640 10 monclient: init 2024-03-25T18:28:34.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:34.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 10 monclient(hunting): picked mon.noname-c con 0x7ff2cc0b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 10 monclient(hunting): picked mon.noname-b con 0x7ff2cc0b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 10 monclient(hunting): picked mon.noname-a con 0x7ff2cc14be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc0af940 con 0x7ff2cc14be90 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc0b0a70 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc0b0d50 con 0x7ff2cc0b1730 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d3cf8640 10 monclient(hunting): authenticate will time out at 7318.476640s 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d2295640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ff2cc14be90 0x7ff2cc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41506/0 (socket says 172.21.15.55:41506) 2024-03-25T18:28:34.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.218+0000 7ff2d2295640 1 -- 172.21.15.55:0/814627140 learned_addr learned my addr 172.21.15.55:0/814627140 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:34.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.221+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749574866 0 0) 0x7ff2cc0b0a70 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.221+0000 7ff2d0a92640 10 monclient(hunting): _negotiate 2024-03-25T18:28:34.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.221+0000 7ff2d0a92640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:34.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.221+0000 7ff2d0a92640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:34.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.221+0000 7ff2d0a92640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:28:34.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.221+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b8003520 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.222+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570526310 0 0) 0x7ff2cc0b0d50 con 0x7ff2cc0b1730 2024-03-25T18:28:34.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.222+0000 7ff2d0a92640 10 monclient(hunting): _negotiate 2024-03-25T18:28:34.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.222+0000 7ff2d0a92640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:34.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.222+0000 7ff2d0a92640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:34.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.222+0000 7ff2d0a92640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:28:34.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.222+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b8003840 con 0x7ff2cc0b1730 2024-03-25T18:28:34.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.224+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923992131 0 0) 0x7ff2cc0af940 con 0x7ff2cc14be90 2024-03-25T18:28:34.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.224+0000 7ff2d0a92640 10 monclient(hunting): _negotiate 2024-03-25T18:28:34.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.224+0000 7ff2d0a92640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:34.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.224+0000 7ff2d0a92640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:34.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.224+0000 7ff2d0a92640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:28:34.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.224+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2cc0b0d50 con 0x7ff2cc14be90 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2603032124 0 0) 0x7ff2b8003520 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 >> v1:172.21.15.55:6790/0 conn(0x7ff2cc0b1730 legacy=0x7ff2cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 >> v1:172.21.15.55:6789/0 conn(0x7ff2cc14be90 legacy=0x7ff2cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 10 monclient: _finish_hunting 0 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 1 monclient: found mon.noname-c 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc0b1030 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 10 monclient: _finish_auth 0 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 10 monclient: _check_auth_tickets 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d3cf8640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:28:34.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d3cf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff2c80015a0 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d0a92640 10 monclient: dump: 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:34.230 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:34.231 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:34.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.227+0000 7ff2d3cf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:34.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.229+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2c8001780 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.229+0000 7ff2d0a92640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:34.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.229+0000 7ff2d0a92640 1 -- 172.21.15.55:0/814627140 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff2c80015a0 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.229+0000 7ff2d3cf8640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:34.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.229+0000 7ff2d3cf8640 4 set_mon_vals no callback set 2024-03-25T18:28:34.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.229+0000 7ff2d0a92640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:34.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.229+0000 7ff2d0a92640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:34.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.229+0000 7ff2d0a92640 10 monclient: dump: 2024-03-25T18:28:34.232 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.230+0000 7ff2d3cf8640 10 monclient: shutdown 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.230+0000 7ff2d3cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.230+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/814627140 >> v1:172.21.15.133:6789/0 conn(0x7ff2cc0b4d60 legacy=0x7ff2cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.230+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/814627140 shutdown_connections 2024-03-25T18:28:34.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.230+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/814627140 >> 172.21.15.55:0/814627140 conn(0x7ff2cc0617a0 msgr2=0x7ff2cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:34.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.231+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/814627140 shutdown_connections 2024-03-25T18:28:34.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.231+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/814627140 wait complete. 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.231+0000 7ff2d3cf8640 10 monclient: build_initial_monmap 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.231+0000 7ff2d3cf8640 10 monclient: monmap: 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.232+0000 7ff2d3cf8640 1 librados: starting msgr at 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.232+0000 7ff2d3cf8640 1 librados: starting objecter 2024-03-25T18:28:34.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.232+0000 7ff2d3cf8640 1 Processor -- start 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.232+0000 7ff2d3cf8640 1 -- start start 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.232+0000 7ff2d3cf8640 1 librados: setting wanted keys 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.232+0000 7ff2d3cf8640 1 librados: calling monclient init 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.232+0000 7ff2d3cf8640 10 monclient: init 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 10 monclient(hunting): picked mon.noname-a con 0x7ff2cc0b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 10 monclient(hunting): picked mon.noname-b con 0x7ff2cc0b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 10 monclient(hunting): picked mon.noname-c con 0x7ff2cc14be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc135c30 con 0x7ff2cc0b1730 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc144d50 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2cc140380 con 0x7ff2cc14be90 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:34.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d3cf8640 10 monclient(hunting): authenticate will time out at 7318.491571s 2024-03-25T18:28:34.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d2295640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7ff2cc14be90 0x7ff2cc1e2270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:38510/0 (socket says 172.21.15.55:38510) 2024-03-25T18:28:34.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.233+0000 7ff2d2295640 1 -- 172.21.15.55:0/3888001600 learned_addr learned my addr 172.21.15.55:0/3888001600 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:34.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.234+0000 7ff2c27fc640 10 client.?.objecter ms_handle_connect 0x7ff2cc14be90 2024-03-25T18:28:34.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.234+0000 7ff2c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:34.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.234+0000 7ff2c27fc640 10 client.?.objecter ms_handle_connect 0x7ff2cc0b1730 2024-03-25T18:28:34.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.234+0000 7ff2c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:34.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.234+0000 7ff2c27fc640 10 client.?.objecter ms_handle_connect 0x7ff2cc0b4d60 2024-03-25T18:28:34.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.234+0000 7ff2c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:34.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.236+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694091269 0 0) 0x7ff2cc135c30 con 0x7ff2cc0b1730 2024-03-25T18:28:34.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.236+0000 7ff2c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:34.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.236+0000 7ff2c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:34.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.236+0000 7ff2c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:34.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.236+0000 7ff2c27fc640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:28:34.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.236+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b0004eb0 con 0x7ff2cc0b1730 2024-03-25T18:28:34.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018858740 0 0) 0x7ff2cc144d50 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:34.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:34.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:34.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:28:34.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b00051d0 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271928750 0 0) 0x7ff2cc140380 con 0x7ff2cc14be90 2024-03-25T18:28:34.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:34.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:34.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:34.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:28:34.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.237+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2cc144d50 con 0x7ff2cc14be90 2024-03-25T18:28:34.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.243+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 809164729 0 0) 0x7ff2b0004eb0 con 0x7ff2cc0b1730 2024-03-25T18:28:34.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.243+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2cc140380 con 0x7ff2cc0b1730 2024-03-25T18:28:34.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.243+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff2c8001560 con 0x7ff2cc0b1730 2024-03-25T18:28:34.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.243+0000 7ff2c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:34.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.243+0000 7ff2c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:34.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.243+0000 7ff2c27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:34.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.243+0000 7ff2c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:34.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.244+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2325275804 0 0) 0x7ff2cc144d50 con 0x7ff2cc14be90 2024-03-25T18:28:34.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.244+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2b0005c80 con 0x7ff2cc14be90 2024-03-25T18:28:34.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.244+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff2c4001bd0 con 0x7ff2cc14be90 2024-03-25T18:28:34.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.244+0000 7ff2c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.244+0000 7ff2c27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.244+0000 7ff2c27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.244+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2936671715 0 0) 0x7ff2b00051d0 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.244+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2cc144d50 con 0x7ff2cc0b4d60 2024-03-25T18:28:34.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1493548327 0 0) 0x7ff2cc140380 con 0x7ff2cc0b1730 2024-03-25T18:28:34.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 >> v1:172.21.15.55:6790/0 conn(0x7ff2cc14be90 legacy=0x7ff2cc1e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:34.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 >> v1:172.21.15.133:6789/0 conn(0x7ff2cc0b4d60 legacy=0x7ff2cc136050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:34.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:34.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 1 monclient: found mon.a 2024-03-25T18:28:34.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc145270 con 0x7ff2cc0b1730 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2d3cf8640 5 monclient: authenticate success, global_id 4224 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2d3cf8640 10 monclient: _renew_subs 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2d3cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2cc140fe0 con 0x7ff2cc0b1730 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2d3cf8640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2d3cf8640 10 monclient: _renew_subs 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2d3cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2cc140fe0 con 0x7ff2cc0b1730 2024-03-25T18:28:34.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.248+0000 7ff2d3cf8640 1 librados: init done 2024-03-25T18:28:34.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.249+0000 7ff2d3cf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:34.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.249+0000 7ff2d3cf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:34.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.249+0000 7ff2d3cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:34.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.249+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2cc140fe0 con 0x7ff2cc0b1730 2024-03-25T18:28:34.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.467+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2c8001100 con 0x7ff2cc0b1730 2024-03-25T18:28:34.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.467+0000 7ff2c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:34.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.467+0000 7ff2c3fff640 4 set_mon_vals no callback set 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.564+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff2c8001560 con 0x7ff2cc0b1730 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.564+0000 7ff2c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.564+0000 7ff2c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.564+0000 7ff2c27fc640 10 monclient: dump: 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:34.566 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:34.567 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:34.567 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:34.567 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:34.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.564+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7ff2c80197d0 con 0x7ff2cc0b1730 2024-03-25T18:28:34.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.565+0000 7ff2d1293640 10 monclient: get_auth_request con 0x7ff2b006e700 auth_method 0 2024-03-25T18:28:34.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.565+0000 7ff2c27fc640 10 client.4224.objecter ms_handle_connect 0x7ff2b006e700 2024-03-25T18:28:34.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.592+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7ff2c80533d0 con 0x7ff2cc0b1730 2024-03-25T18:28:34.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.592+0000 7ff2c27fc640 10 client.4224.objecter ms_dispatch 0x7ff2cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:34.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.592+0000 7ff2c27fc640 3 client.4224.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:34.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.592+0000 7ff2c27fc640 3 client.4224.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:34.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.592+0000 7ff2c27fc640 20 client.4224.objecter dump_active .. 0 homeless 2024-03-25T18:28:34.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.592+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2c804ed10 con 0x7ff2cc0b1730 2024-03-25T18:28:34.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.592+0000 7ff2c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:34.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.592+0000 7ff2c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:34.596 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:16.805 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:34.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.778+0000 7ff2d3cf8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:34.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.778+0000 7ff2d3cf8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:34.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.778+0000 7ff2d3cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:34.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.778+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2cc140fe0 con 0x7ff2cc0b1730 2024-03-25T18:28:34.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.795+0000 7ff2c27fc640 1 -- 172.21.15.55:0/3888001600 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7ff2c8050e10 con 0x7ff2cc0b1730 2024-03-25T18:28:34.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.795+0000 7ff2c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:34.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.795+0000 7ff2c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:34.797 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:28:34.797 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 10 librados: watch_flush enter 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 10 librados: watch_flush exit 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 20 client.4224.objecter shutdown clearing up homeless session... 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 10 client.4224.objecter successfully canceled tick 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 >> v1:172.21.15.55:6800/286536064 conn(0x7ff2b006e700 legacy=0x7ff2b0070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 10 monclient: shutdown 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 >> v1:172.21.15.55:6789/0 conn(0x7ff2cc0b1730 legacy=0x7ff2cc141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 shutdown_connections 2024-03-25T18:28:34.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.797+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 >> 172.21.15.55:0/3888001600 conn(0x7ff2cc0617a0 msgr2=0x7ff2cc0b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:34.801 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.798+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 shutdown_connections 2024-03-25T18:28:34.801 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.798+0000 7ff2d3cf8640 1 -- 172.21.15.55:0/3888001600 wait complete. 2024-03-25T18:28:34.801 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:34.798+0000 7ff2d3cf8640 1 librados: shutdown 2024-03-25T18:28:34.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:35.343 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:17.527 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:35.409 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:17.609 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:35.432 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:17.627 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:35.480 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:17.700 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:35.666 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:17.861 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:38.926 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:21.138 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:38.932 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:21.144 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:39.596 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:21.800 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:39.600 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:21.804 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:39.848 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:22.056 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:39.854 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:22.063 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:40.564 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:22.747 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:40.569 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:22.753 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:41.009 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:23.203 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:41.013 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:23.208 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:41.149 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:23.349 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:41.153 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:23.353 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:41.393 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:23.613 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:41.399 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:23.619 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:41.438 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:23.633 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:41.442 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:23.637 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:41.616 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:41.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.707+0000 7f79f915d640 10 monclient: get_monmap_and_config 2024-03-25T18:28:41.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.707+0000 7f79f915d640 10 monclient: build_initial_monmap 2024-03-25T18:28:41.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.707+0000 7f79f915d640 10 monclient: monmap: 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:28:41.708562+0000 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:28:41.708562+0000 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:41.709 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.708+0000 7f79f915d640 1 Processor -- start 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.708+0000 7f79f915d640 1 -- start start 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.708+0000 7f79f915d640 10 monclient: init 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.708+0000 7f79f915d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.708+0000 7f79f915d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.708+0000 7f79f915d640 10 monclient(hunting): picked mon.noname-c con 0x7f79f40b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.708+0000 7f79f915d640 10 monclient(hunting): picked mon.noname-a con 0x7f79f40b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.709+0000 7f79f915d640 10 monclient(hunting): picked mon.noname-b con 0x7f79f414be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.709+0000 7f79f915d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f40af940 con 0x7f79f40b1730 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.709+0000 7f79f915d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f40b0a70 con 0x7f79f40b4d60 2024-03-25T18:28:41.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.709+0000 7f79f915d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f40b0d50 con 0x7f79f414be90 2024-03-25T18:28:41.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.709+0000 7f79f915d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:41.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.709+0000 7f79f915d640 10 monclient(hunting): authenticate will time out at 7325.966408s 2024-03-25T18:28:41.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.709+0000 7f79f359e640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f79f414be90 0x7f79f414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:41432/0 (socket says 172.21.15.55:41432) 2024-03-25T18:28:41.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.709+0000 7f79f359e640 1 -- 172.21.15.55:0/2951562041 learned_addr learned my addr 172.21.15.55:0/2951562041 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:41.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142006366 0 0) 0x7f79f40b0d50 con 0x7f79f414be90 2024-03-25T18:28:41.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:28:41.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79dc003520 con 0x7f79f414be90 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888072213 0 0) 0x7f79f40af940 con 0x7f79f40b1730 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 10 monclient(hunting): my global_id is 4227 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.712+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f40b0d50 con 0x7f79f40b1730 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.713+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959631457 0 0) 0x7f79f40b0a70 con 0x7f79f40b4d60 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.713+0000 7f79f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.713+0000 7f79f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.713+0000 7f79f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.713+0000 7f79f1d9b640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:28:41.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.713+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f40af940 con 0x7f79f40b4d60 2024-03-25T18:28:41.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2851834484 0 0) 0x7f79dc003520 con 0x7f79f414be90 2024-03-25T18:28:41.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 >> v1:172.21.15.133:6789/0 conn(0x7f79f40b4d60 legacy=0x7f79f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 >> v1:172.21.15.55:6789/0 conn(0x7f79f40b1730 legacy=0x7f79f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 1 monclient: found mon.noname-b 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f40b1030 con 0x7f79f414be90 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f915d640 5 monclient: authenticate success, global_id 4226 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f915d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f79ec0019d0 con 0x7f79f414be90 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f1d9b640 10 monclient: dump: 2024-03-25T18:28:41.718 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:41.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.715+0000 7f79f915d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:41.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.717+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79ec001c20 con 0x7f79f414be90 2024-03-25T18:28:41.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.717+0000 7f79f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:41.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.717+0000 7f79f1d9b640 1 -- 172.21.15.55:0/2951562041 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f79ec001fb0 con 0x7f79f414be90 2024-03-25T18:28:41.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.717+0000 7f79f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.717+0000 7f79f1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.717+0000 7f79f1d9b640 10 monclient: dump: 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.717+0000 7f79f915d640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.717+0000 7f79f915d640 4 set_mon_vals no callback set 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.718+0000 7f79f915d640 10 monclient: shutdown 2024-03-25T18:28:41.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.718+0000 7f79f915d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.718+0000 7f79f915d640 1 -- 172.21.15.55:0/2951562041 >> v1:172.21.15.55:6790/0 conn(0x7f79f414be90 legacy=0x7f79f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.718+0000 7f79f915d640 1 -- 172.21.15.55:0/2951562041 shutdown_connections 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.718+0000 7f79f915d640 1 -- 172.21.15.55:0/2951562041 >> 172.21.15.55:0/2951562041 conn(0x7f79f40617a0 msgr2=0x7f79f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.718+0000 7f79f915d640 1 -- 172.21.15.55:0/2951562041 shutdown_connections 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.718+0000 7f79f915d640 1 -- 172.21.15.55:0/2951562041 wait complete. 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.719+0000 7f79f915d640 10 monclient: build_initial_monmap 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.719+0000 7f79f915d640 10 monclient: monmap: 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:28:41.725 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.720+0000 7f79f915d640 1 librados: starting msgr at 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.720+0000 7f79f915d640 1 librados: starting objecter 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.720+0000 7f79f915d640 1 Processor -- start 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.720+0000 7f79f915d640 1 -- start start 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.720+0000 7f79f915d640 1 librados: setting wanted keys 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.720+0000 7f79f915d640 1 librados: calling monclient init 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.720+0000 7f79f915d640 10 monclient: init 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f915d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f915d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f915d640 10 monclient(hunting): picked mon.noname-c con 0x7f79f40b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:41.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f915d640 10 monclient(hunting): picked mon.noname-b con 0x7f79f40b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f915d640 10 monclient(hunting): picked mon.noname-a con 0x7f79f414be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f915d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f4140d00 con 0x7f79f414be90 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f915d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f4141f10 con 0x7f79f40b4d60 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f2d9d640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f79f40b1730 0x7f79f4138080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:41436/0 (socket says 172.21.15.55:41436) 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f2d9d640 1 -- 172.21.15.55:0/1458565778 learned_addr learned my addr 172.21.15.55:0/1458565778 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.721+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f41d7a20 con 0x7f79f40b1730 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.722+0000 7f79f915d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.722+0000 7f79f915d640 10 monclient(hunting): authenticate will time out at 7325.979439s 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.722+0000 7f79db7fe640 10 client.?.objecter ms_handle_connect 0x7f79f40b1730 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.722+0000 7f79db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.722+0000 7f79db7fe640 10 client.?.objecter ms_handle_connect 0x7f79f40b4d60 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.722+0000 7f79db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.722+0000 7f79db7fe640 10 client.?.objecter ms_handle_connect 0x7f79f414be90 2024-03-25T18:28:41.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.722+0000 7f79db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253329756 0 0) 0x7f79f4140d00 con 0x7f79f414be90 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): my global_id is 4230 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e0004eb0 con 0x7f79f414be90 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159257633 0 0) 0x7f79f41d7a20 con 0x7f79f40b1730 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): my global_id is 4229 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f4140d00 con 0x7f79f40b1730 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465719143 0 0) 0x7f79f4141f10 con 0x7f79f40b4d60 2024-03-25T18:28:41.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:28:41.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.724+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e00053b0 con 0x7f79f40b4d60 2024-03-25T18:28:41.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3783655174 0 0) 0x7f79f4140d00 con 0x7f79f40b1730 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79f4141f10 con 0x7f79f40b1730 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f79e4001a30 con 0x7f79f40b1730 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:41.732 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2858788094 0 0) 0x7f79e0004eb0 con 0x7f79f414be90 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79e0005d70 con 0x7f79f414be90 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f79ec001f30 con 0x7f79f414be90 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.729+0000 7f79db7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:41.733 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:41.734 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:41.734 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:41.734 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.731+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 879958053 0 0) 0x7f79e00053b0 con 0x7f79f40b4d60 2024-03-25T18:28:41.734 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.731+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79e0004eb0 con 0x7f79f40b4d60 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.731+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f79e80018f0 con 0x7f79f40b4d60 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.731+0000 7f79db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.731+0000 7f79db7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.731+0000 7f79db7fe640 10 monclient(hunting): dump: 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:41.737 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3174836736 0 0) 0x7f79e0005d70 con 0x7f79f414be90 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 >> v1:172.21.15.55:6790/0 conn(0x7f79f40b1730 legacy=0x7f79f4138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 >> v1:172.21.15.133:6789/0 conn(0x7f79f40b4d60 legacy=0x7f79f41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 1 monclient: found mon.a 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f41d8650 con 0x7f79f414be90 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:41.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79f915d640 5 monclient: authenticate success, global_id 4230 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79f915d640 10 monclient: _renew_subs 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79f915d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f41422c0 con 0x7f79f414be90 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79f915d640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79f915d640 10 monclient: _renew_subs 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79f915d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f4142ae0 con 0x7f79f414be90 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.733+0000 7f79f915d640 1 librados: init done 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.734+0000 7f79f915d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.735+0000 7f79f915d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.735+0000 7f79f915d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:41.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.735+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f41d7bb0 con 0x7f79f414be90 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.750+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79ec002090 con 0x7f79f414be90 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.750+0000 7f79db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.750+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f79ec004e10 con 0x7f79f414be90 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.750+0000 7f79db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.750+0000 7f79db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.750+0000 7f79db7fe640 10 monclient: dump: 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:41.751 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:41.752 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:41.752 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:41.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.750+0000 7f79f159a640 4 set_mon_vals no callback set 2024-03-25T18:28:41.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.750+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f79ec01ade0 con 0x7f79f414be90 2024-03-25T18:28:41.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.751+0000 7f79f2d9d640 10 monclient: get_auth_request con 0x7f79e006e630 auth_method 0 2024-03-25T18:28:41.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.752+0000 7f79db7fe640 10 client.4230.objecter ms_handle_connect 0x7f79e006e630 2024-03-25T18:28:41.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.753+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f79ec054920 con 0x7f79f414be90 2024-03-25T18:28:41.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.753+0000 7f79db7fe640 10 client.4230.objecter ms_dispatch 0x7f79f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:41.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.753+0000 7f79db7fe640 3 client.4230.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:41.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.753+0000 7f79db7fe640 3 client.4230.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:41.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:41.753+0000 7f79db7fe640 20 client.4230.objecter dump_active .. 0 homeless 2024-03-25T18:28:42.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.076+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79ec050260 con 0x7f79f414be90 2024-03-25T18:28:42.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.076+0000 7f79db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:42.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.076+0000 7f79db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:42.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.344+0000 7f79f915d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:42.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.344+0000 7f79f915d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:42.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.344+0000 7f79f915d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:42.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.344+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79f40627b0 con 0x7f79f414be90 2024-03-25T18:28:42.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.370+0000 7f79db7fe640 1 -- 172.21.15.55:0/1458565778 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f79ec052360 con 0x7f79f414be90 2024-03-25T18:28:42.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.370+0000 7f79db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:42.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.370+0000 7f79db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:42.371 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:28:42.371 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:42.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 10 librados: watch_flush enter 2024-03-25T18:28:42.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 10 librados: watch_flush exit 2024-03-25T18:28:42.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 20 client.4230.objecter shutdown clearing up homeless session... 2024-03-25T18:28:42.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 10 client.4230.objecter successfully canceled tick 2024-03-25T18:28:42.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 >> v1:172.21.15.55:6800/286536064 conn(0x7f79e006e630 legacy=0x7f79e0070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:42.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 10 monclient: shutdown 2024-03-25T18:28:42.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:42.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 >> v1:172.21.15.55:6789/0 conn(0x7f79f414be90 legacy=0x7f79f41d7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:42.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 shutdown_connections 2024-03-25T18:28:42.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 >> 172.21.15.55:0/1458565778 conn(0x7f79f40617a0 msgr2=0x7f79f40b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:42.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 shutdown_connections 2024-03-25T18:28:42.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 1 -- 172.21.15.55:0/1458565778 wait complete. 2024-03-25T18:28:42.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:42.372+0000 7f79f915d640 1 librados: shutdown 2024-03-25T18:28:42.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:43.326 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:00:25.538 35430== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:43.865 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:00:26.069 35432== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:44.154 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:00:26.363 35431== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:44.985 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:00:27.169 35433== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:45.426 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:00:27.621 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:45.479 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:00:27.680 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:45.715 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:00:27.910 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:45.811 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:00:28.031 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:28:49.190 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.294+0000 7f1dce2a5640 10 monclient: get_monmap_and_config 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.294+0000 7f1dce2a5640 10 monclient: build_initial_monmap 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.294+0000 7f1dce2a5640 10 monclient: monmap: 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:28:49.295257+0000 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:28:49.295257+0000 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:49.295 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:49.296 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:49.296 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:49.296 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.295+0000 7f1dce2a5640 1 Processor -- start 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.295+0000 7f1dce2a5640 1 -- start start 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.295+0000 7f1dce2a5640 10 monclient: init 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.295+0000 7f1dce2a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.295+0000 7f1dce2a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.295+0000 7f1dce2a5640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc80b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.295+0000 7f1dce2a5640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc80b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.295+0000 7f1dce2a5640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc814be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.295+0000 7f1dce2a5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc80af940 con 0x7f1dc80b4d60 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.296+0000 7f1dce2a5640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc80b0a70 con 0x7f1dc814be90 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.296+0000 7f1dce2a5640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc80b0d50 con 0x7f1dc80b1730 2024-03-25T18:28:49.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.296+0000 7f1dce2a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:49.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.296+0000 7f1dce2a5640 10 monclient(hunting): authenticate will time out at 7333.553406s 2024-03-25T18:28:49.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.296+0000 7f1dc77fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f1dc80b1730 0x7f1dc80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:53914/0 (socket says 172.21.15.55:53914) 2024-03-25T18:28:49.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.296+0000 7f1dc77fe640 1 -- 172.21.15.55:0/935051884 learned_addr learned my addr 172.21.15.55:0/935051884 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:49.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.325+0000 7f1dc6ffd640 1 -- 172.21.15.55:0/935051884 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660289306 0 0) 0x7f1dc80af940 con 0x7f1dc80b4d60 2024-03-25T18:28:49.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.325+0000 7f1dc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:49.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.325+0000 7f1dc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:49.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.325+0000 7f1dc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:49.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.325+0000 7f1dc6ffd640 10 monclient(hunting): my global_id is 4233 2024-03-25T18:28:49.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.325+0000 7f1dc6ffd640 1 -- 172.21.15.55:0/935051884 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db0003520 con 0x7f1dc80b4d60 2024-03-25T18:28:49.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 1 -- 172.21.15.55:0/935051884 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2008938007 0 0) 0x7f1db0003520 con 0x7f1dc80b4d60 2024-03-25T18:28:49.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 1 -- 172.21.15.55:0/935051884 >> v1:172.21.15.55:6790/0 conn(0x7f1dc80b1730 legacy=0x7f1dc80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 1 -- 172.21.15.55:0/935051884 >> v1:172.21.15.133:6789/0 conn(0x7f1dc814be90 legacy=0x7f1dc814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:49.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:28:49.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:49.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 1 -- 172.21.15.55:0/935051884 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc80b1030 con 0x7f1dc80b4d60 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 1 -- 172.21.15.55:0/935051884 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1db8001690 con 0x7f1dc80b4d60 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dc6ffd640 10 monclient: dump: 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:49.332 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:49.333 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:49.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dce2a5640 5 monclient: authenticate success, global_id 4233 2024-03-25T18:28:49.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.330+0000 7f1dce2a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:49.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.332+0000 7f1dc6ffd640 1 -- 172.21.15.55:0/935051884 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1db80017f0 con 0x7f1dc80b4d60 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.332+0000 7f1dc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.332+0000 7f1dc6ffd640 1 -- 172.21.15.55:0/935051884 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1db8001c40 con 0x7f1dc80b4d60 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.332+0000 7f1dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.332+0000 7f1dc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.332+0000 7f1dc6ffd640 10 monclient: dump: 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:49.334 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.332+0000 7f1dce2a5640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.332+0000 7f1dce2a5640 4 set_mon_vals no callback set 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.333+0000 7f1dce2a5640 10 monclient: shutdown 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.333+0000 7f1dce2a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.333+0000 7f1dce2a5640 1 -- 172.21.15.55:0/935051884 >> v1:172.21.15.55:6789/0 conn(0x7f1dc80b4d60 legacy=0x7f1dc813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.333+0000 7f1dce2a5640 1 -- 172.21.15.55:0/935051884 shutdown_connections 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.333+0000 7f1dce2a5640 1 -- 172.21.15.55:0/935051884 >> 172.21.15.55:0/935051884 conn(0x7f1dc80617a0 msgr2=0x7f1dc8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.333+0000 7f1dce2a5640 1 -- 172.21.15.55:0/935051884 shutdown_connections 2024-03-25T18:28:49.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.333+0000 7f1dce2a5640 1 -- 172.21.15.55:0/935051884 wait complete. 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.333+0000 7f1dce2a5640 10 monclient: build_initial_monmap 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.333+0000 7f1dce2a5640 10 monclient: monmap: 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:49.337 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.334+0000 7f1dce2a5640 1 librados: starting msgr at 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.334+0000 7f1dce2a5640 1 librados: starting objecter 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.334+0000 7f1dce2a5640 1 Processor -- start 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.334+0000 7f1dce2a5640 1 -- start start 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.334+0000 7f1dce2a5640 1 librados: setting wanted keys 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.334+0000 7f1dce2a5640 1 librados: calling monclient init 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.334+0000 7f1dce2a5640 10 monclient: init 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc80b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc80b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc814be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc8142f70 con 0x7f1dc80b1730 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc81411b0 con 0x7f1dc814be90 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc8143250 con 0x7f1dc80b4d60 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.335+0000 7f1dce2a5640 10 monclient(hunting): authenticate will time out at 7333.592873s 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.336+0000 7f1dc7fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1dc80b1730 0x7f1dc81372b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45810/0 (socket says 172.21.15.55:45810) 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.336+0000 7f1dc7fff640 1 -- 172.21.15.55:0/1170938875 learned_addr learned my addr 172.21.15.55:0/1170938875 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.336+0000 7f1dc4ff9640 10 client.?.objecter ms_handle_connect 0x7f1dc80b1730 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.336+0000 7f1dc4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:49.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.339+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921554915 0 0) 0x7f1dc8142f70 con 0x7f1dc80b1730 2024-03-25T18:28:49.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.339+0000 7f1dc4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:49.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.339+0000 7f1dc4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:49.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.339+0000 7f1dc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:49.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.339+0000 7f1dc4ff9640 10 monclient(hunting): my global_id is 4236 2024-03-25T18:28:49.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.339+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db4004eb0 con 0x7f1dc80b1730 2024-03-25T18:28:49.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.346+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 246267317 0 0) 0x7f1db4004eb0 con 0x7f1dc80b1730 2024-03-25T18:28:49.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.346+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dc8142f70 con 0x7f1dc80b1730 2024-03-25T18:28:49.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.346+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1db8001690 con 0x7f1dc80b1730 2024-03-25T18:28:49.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.346+0000 7f1dc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.346+0000 7f1dc4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:49.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.346+0000 7f1dc4ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:49.347 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:49.347 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:49.348 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:49.348 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:49.348 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.348 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:49.348 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:49.348 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:49.348 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:49.348 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:49.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.346+0000 7f1dc4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:49.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3859370364 0 0) 0x7f1dc8142f70 con 0x7f1dc80b1730 2024-03-25T18:28:49.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 >> v1:172.21.15.55:6790/0 conn(0x7f1dc80b4d60 legacy=0x7f1dc813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 >> v1:172.21.15.133:6789/0 conn(0x7f1dc814be90 legacy=0x7f1dc8141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:49.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 1 monclient: found mon.a 2024-03-25T18:28:49.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:49.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:49.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc8140c90 con 0x7f1dc80b1730 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.350+0000 7f1dc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.351+0000 7f1dce2a5640 5 monclient: authenticate success, global_id 4236 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.351+0000 7f1dce2a5640 10 monclient: _renew_subs 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.351+0000 7f1dce2a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.351+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc81379c0 con 0x7f1dc80b1730 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.351+0000 7f1dce2a5640 10 client.4236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.351+0000 7f1dce2a5640 10 monclient: _renew_subs 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.351+0000 7f1dce2a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.351+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc813d9e0 con 0x7f1dc80b1730 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.351+0000 7f1dce2a5640 1 librados: init done 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.352+0000 7f1dce2a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:49.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.352+0000 7f1dce2a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:49.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.352+0000 7f1dce2a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:49.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.352+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc80627b0 con 0x7f1dc80b1730 2024-03-25T18:28:49.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.477+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1db8003960 con 0x7f1dc80b1730 2024-03-25T18:28:49.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.477+0000 7f1dc4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:49.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.477+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1db8003db0 con 0x7f1dc80b1730 2024-03-25T18:28:49.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.477+0000 7f1dc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.477+0000 7f1dc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:49.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.477+0000 7f1dc4ff9640 10 monclient: dump: 2024-03-25T18:28:49.479 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:49.479 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:49.479 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.477+0000 7f1dc67fc640 4 set_mon_vals no callback set 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.477+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f1db8003fd0 con 0x7f1dc80b1730 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.478+0000 7f1dc77fe640 10 monclient: get_auth_request con 0x7f1db406db30 auth_method 0 2024-03-25T18:28:49.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.478+0000 7f1dc4ff9640 10 client.4236.objecter ms_handle_connect 0x7f1db406db30 2024-03-25T18:28:49.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.661+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f1db8054bc0 con 0x7f1dc80b1730 2024-03-25T18:28:49.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.661+0000 7f1dc4ff9640 10 client.4236.objecter ms_dispatch 0x7f1dc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:49.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.661+0000 7f1dc4ff9640 3 client.4236.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:49.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.661+0000 7f1dc4ff9640 3 client.4236.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:49.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.662+0000 7f1dc4ff9640 20 client.4236.objecter dump_active .. 0 homeless 2024-03-25T18:28:49.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.662+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1db80506d0 con 0x7f1dc80b1730 2024-03-25T18:28:49.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.662+0000 7f1dc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:49.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.662+0000 7f1dc4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:49.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.908+0000 7f1dce2a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:49.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.908+0000 7f1dce2a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:49.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.908+0000 7f1dce2a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:49.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.908+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dc814bbf0 con 0x7f1dc80b1730 2024-03-25T18:28:49.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.934+0000 7f1dc4ff9640 1 -- 172.21.15.55:0/1170938875 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f1db8052600 con 0x7f1dc80b1730 2024-03-25T18:28:49.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.934+0000 7f1dc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:49.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.934+0000 7f1dc4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:49.936 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:28:49.936 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:49.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.936+0000 7f1dce2a5640 10 librados: watch_flush enter 2024-03-25T18:28:49.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.936+0000 7f1dce2a5640 10 librados: watch_flush exit 2024-03-25T18:28:49.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.936+0000 7f1dce2a5640 20 client.4236.objecter shutdown clearing up homeless session... 2024-03-25T18:28:49.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.936+0000 7f1dce2a5640 10 client.4236.objecter successfully canceled tick 2024-03-25T18:28:49.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.936+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 >> v1:172.21.15.55:6800/286536064 conn(0x7f1db406db30 legacy=0x7f1db406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.936+0000 7f1dce2a5640 10 monclient: shutdown 2024-03-25T18:28:49.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.936+0000 7f1dce2a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:49.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.936+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 >> v1:172.21.15.55:6789/0 conn(0x7f1dc80b1730 legacy=0x7f1dc81372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.937+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 shutdown_connections 2024-03-25T18:28:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.937+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 >> 172.21.15.55:0/1170938875 conn(0x7f1dc80617a0 msgr2=0x7f1dc80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.938+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 shutdown_connections 2024-03-25T18:28:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.938+0000 7f1dce2a5640 1 -- 172.21.15.55:0/1170938875 wait complete. 2024-03-25T18:28:49.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:49.938+0000 7f1dce2a5640 1 librados: shutdown 2024-03-25T18:28:49.949 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:51.679 INFO:tasks.ceph.osd.4.smithi133.stderr:2024-03-25T18:28:51.675+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:28:52.035 INFO:tasks.ceph.osd.6.smithi133.stderr:2024-03-25T18:28:52.024+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:28:52.279 INFO:tasks.ceph.osd.5.smithi133.stderr:2024-03-25T18:28:52.274+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:28:53.801 INFO:tasks.ceph.osd.7.smithi133.stderr:2024-03-25T18:28:53.796+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:28:54.700 INFO:tasks.ceph.osd.3.smithi055.stderr:2024-03-25T18:28:54.696+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:28:54.854 INFO:tasks.ceph.osd.2.smithi055.stderr:2024-03-25T18:28:54.849+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:28:54.990 INFO:tasks.ceph.osd.1.smithi055.stderr:2024-03-25T18:28:54.986+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:28:55.311 INFO:tasks.ceph.osd.0.smithi055.stderr:2024-03-25T18:28:55.306+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:28:56.754 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:56.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.888+0000 7f1c64d2b640 10 monclient: get_monmap_and_config 2024-03-25T18:28:56.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.888+0000 7f1c64d2b640 10 monclient: build_initial_monmap 2024-03-25T18:28:56.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.888+0000 7f1c64d2b640 10 monclient: monmap: 2024-03-25T18:28:56.889 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:56.889 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:56.889 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:28:56.889363+0000 2024-03-25T18:28:56.890 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:28:56.889363+0000 2024-03-25T18:28:56.890 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:56.890 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:56.890 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:56.890 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:28:56.890 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:28:56.890 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.889+0000 7f1c64d2b640 1 Processor -- start 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.889+0000 7f1c64d2b640 1 -- start start 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.889+0000 7f1c64d2b640 10 monclient: init 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f1c600b1ef0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f1c601473a0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f1c60150070 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c600b2570 con 0x7f1c601473a0 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c60146c20 con 0x7f1c60150070 2024-03-25T18:28:56.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c60146e80 con 0x7f1c600b1ef0 2024-03-25T18:28:56.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:56.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.890+0000 7f1c64d2b640 10 monclient(hunting): authenticate will time out at 7341.147962s 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.897+0000 7f1c5dd9b640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1c601473a0 0x7f1c60143b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46118/0 (socket says 172.21.15.55:46118) 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.897+0000 7f1c5dd9b640 1 -- 172.21.15.55:0/2999140076 learned_addr learned my addr 172.21.15.55:0/2999140076 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.903+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609262606 0 0) 0x7f1c600b2570 con 0x7f1c601473a0 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.903+0000 7f1c5d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.903+0000 7f1c5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.903+0000 7f1c5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.903+0000 7f1c5d59a640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.903+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c48003520 con 0x7f1c601473a0 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.907+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386816833 0 0) 0x7f1c60146c20 con 0x7f1c60150070 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.907+0000 7f1c5d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.907+0000 7f1c5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.907+0000 7f1c5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.907+0000 7f1c5d59a640 10 monclient(hunting): my global_id is 4243 2024-03-25T18:28:56.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.907+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c600b2570 con 0x7f1c60150070 2024-03-25T18:28:56.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 82180327 0 0) 0x7f1c48003520 con 0x7f1c601473a0 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 >> v1:172.21.15.55:6790/0 conn(0x7f1c600b1ef0 legacy=0x7f1c600b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 >> v1:172.21.15.133:6789/0 conn(0x7f1c60150070 legacy=0x7f1c60152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 1 monclient: found mon.noname-a 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6006e1b0 con 0x7f1c601473a0 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 10 monclient: _finish_auth 0 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c50001690 con 0x7f1c601473a0 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:56.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c5d59a640 10 monclient: dump: 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c64d2b640 5 monclient: authenticate success, global_id 4263 2024-03-25T18:28:56.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.914+0000 7f1c64d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:56.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c500017f0 con 0x7f1c601473a0 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c5d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c5d59a640 1 -- 172.21.15.55:0/2999140076 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c50001c00 con 0x7f1c601473a0 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c64d2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c64d2b640 4 set_mon_vals no callback set 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c5d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c5d59a640 10 monclient: dump: 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:56.928 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:56.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c64d2b640 10 monclient: shutdown 2024-03-25T18:28:56.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c64d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:56.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.927+0000 7f1c64d2b640 1 -- 172.21.15.55:0/2999140076 >> v1:172.21.15.55:6789/0 conn(0x7f1c601473a0 legacy=0x7f1c60143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.928+0000 7f1c64d2b640 1 -- 172.21.15.55:0/2999140076 shutdown_connections 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.928+0000 7f1c64d2b640 1 -- 172.21.15.55:0/2999140076 >> 172.21.15.55:0/2999140076 conn(0x7f1c600b7ea0 msgr2=0x7f1c600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.928+0000 7f1c64d2b640 1 -- 172.21.15.55:0/2999140076 shutdown_connections 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.928+0000 7f1c64d2b640 1 -- 172.21.15.55:0/2999140076 wait complete. 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.929+0000 7f1c64d2b640 10 monclient: build_initial_monmap 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.929+0000 7f1c64d2b640 10 monclient: monmap: 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:28:56.931 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:56.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.929+0000 7f1c64d2b640 1 librados: starting msgr at 2024-03-25T18:28:56.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.929+0000 7f1c64d2b640 1 librados: starting objecter 2024-03-25T18:28:56.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.929+0000 7f1c64d2b640 1 Processor -- start 2024-03-25T18:28:56.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.929+0000 7f1c64d2b640 1 -- start start 2024-03-25T18:28:56.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.929+0000 7f1c64d2b640 1 librados: setting wanted keys 2024-03-25T18:28:56.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.929+0000 7f1c64d2b640 1 librados: calling monclient init 2024-03-25T18:28:56.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.929+0000 7f1c64d2b640 10 monclient: init 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f1c600b1ef0 addr v1:172.21.15.55:6790/0 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f1c601473a0 addr v1:172.21.15.55:6789/0 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f1c60150070 addr v1:172.21.15.133:6789/0 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c600b34f0 con 0x7f1c601473a0 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c600b5080 con 0x7f1c60150070 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c600b4170 con 0x7f1c600b1ef0 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:56.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.930+0000 7f1c64d2b640 10 monclient(hunting): authenticate will time out at 7341.187843s 2024-03-25T18:28:56.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.935+0000 7f1c5ed9d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f1c60150070 0x7f1c600b3980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:56582/0 (socket says 172.21.15.55:56582) 2024-03-25T18:28:56.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.935+0000 7f1c5ed9d640 1 -- 172.21.15.55:0/953272116 learned_addr learned my addr 172.21.15.55:0/953272116 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:28:56.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.936+0000 7f1c4effd640 10 client.?.objecter ms_handle_connect 0x7f1c60150070 2024-03-25T18:28:56.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.936+0000 7f1c4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:56.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.942+0000 7f1c4effd640 10 client.?.objecter ms_handle_connect 0x7f1c601473a0 2024-03-25T18:28:56.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.942+0000 7f1c4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:56.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.947+0000 7f1c4effd640 10 client.?.objecter ms_handle_connect 0x7f1c600b1ef0 2024-03-25T18:28:56.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.947+0000 7f1c4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:56.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.950+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638879920 0 0) 0x7f1c600b34f0 con 0x7f1c601473a0 2024-03-25T18:28:56.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.950+0000 7f1c4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.950+0000 7f1c4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.950+0000 7f1c4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.950+0000 7f1c4effd640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:28:56.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.950+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c40004eb0 con 0x7f1c601473a0 2024-03-25T18:28:56.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.952+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588807420 0 0) 0x7f1c600b5080 con 0x7f1c60150070 2024-03-25T18:28:56.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.952+0000 7f1c4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.952+0000 7f1c4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.952+0000 7f1c4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.952+0000 7f1c4effd640 10 monclient(hunting): my global_id is 4246 2024-03-25T18:28:56.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.952+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c600b34f0 con 0x7f1c60150070 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.958+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 133287915 0 0) 0x7f1c40004eb0 con 0x7f1c601473a0 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.958+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c600b5080 con 0x7f1c601473a0 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.958+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c50001b00 con 0x7f1c601473a0 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.958+0000 7f1c4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.958+0000 7f1c4effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.958+0000 7f1c4effd640 10 monclient(hunting): dump: 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.958+0000 7f1c4effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:56.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.963+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079633042 0 0) 0x7f1c600b4170 con 0x7f1c600b1ef0 2024-03-25T18:28:56.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.963+0000 7f1c4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.963+0000 7f1c4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.963+0000 7f1c4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.963+0000 7f1c4effd640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:28:56.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.963+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c40004eb0 con 0x7f1c600b1ef0 2024-03-25T18:28:56.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.967+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270112814 0 0) 0x7f1c600b34f0 con 0x7f1c60150070 2024-03-25T18:28:56.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.967+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c600b4170 con 0x7f1c60150070 2024-03-25T18:28:56.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.967+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c58001c10 con 0x7f1c60150070 2024-03-25T18:28:56.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.967+0000 7f1c4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.967+0000 7f1c4effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:56.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.967+0000 7f1c4effd640 10 monclient(hunting): dump: 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:56.969 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:56.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 998444431 0 0) 0x7f1c600b5080 con 0x7f1c601473a0 2024-03-25T18:28:56.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 >> v1:172.21.15.55:6790/0 conn(0x7f1c600b1ef0 legacy=0x7f1c60139160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 >> v1:172.21.15.133:6789/0 conn(0x7f1c60150070 legacy=0x7f1c600b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 1 monclient: found mon.a 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c600b68f0 con 0x7f1c601473a0 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 10 monclient: _finish_auth 0 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c64d2b640 5 monclient: authenticate success, global_id 4266 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c64d2b640 10 monclient: _renew_subs 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c64d2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c600b6ba0 con 0x7f1c601473a0 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c64d2b640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c64d2b640 10 monclient: _renew_subs 2024-03-25T18:28:56.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c64d2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:56.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c601e7b10 con 0x7f1c601473a0 2024-03-25T18:28:56.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.969+0000 7f1c64d2b640 1 librados: init done 2024-03-25T18:28:56.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.970+0000 7f1c64d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:56.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.970+0000 7f1c64d2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:56.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.970+0000 7f1c64d2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:56.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:56.970+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c600b36a0 con 0x7f1c601473a0 2024-03-25T18:28:57.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.290+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c50004a90 con 0x7f1c601473a0 2024-03-25T18:28:57.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.290+0000 7f1c4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:57.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.290+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1c50004c00 con 0x7f1c601473a0 2024-03-25T18:28:57.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.290+0000 7f1c4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:57.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.290+0000 7f1c4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.290+0000 7f1c4effd640 10 monclient: dump: 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:28:57.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.290+0000 7f1c5cd99640 4 set_mon_vals no callback set 2024-03-25T18:28:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.291+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f1c5001abd0 con 0x7f1c601473a0 2024-03-25T18:28:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.292+0000 7f1c5e59c640 10 monclient: get_auth_request con 0x7f1c4006dfb0 auth_method 0 2024-03-25T18:28:57.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.292+0000 7f1c4effd640 10 client.4266.objecter ms_handle_connect 0x7f1c4006dfb0 2024-03-25T18:28:57.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.297+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f1c50054680 con 0x7f1c601473a0 2024-03-25T18:28:57.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.297+0000 7f1c4effd640 10 client.4266.objecter ms_dispatch 0x7f1c60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:57.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.297+0000 7f1c4effd640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:57.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.297+0000 7f1c4effd640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:57.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.298+0000 7f1c4effd640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-25T18:28:57.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.298+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c50050190 con 0x7f1c601473a0 2024-03-25T18:28:57.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.298+0000 7f1c4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:57.301 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.298+0000 7f1c4effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:57.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.551+0000 7f1c64d2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:57.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.551+0000 7f1c64d2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:57.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.551+0000 7f1c64d2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:28:57.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.551+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c600627b0 con 0x7f1c601473a0 2024-03-25T18:28:57.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.615+0000 7f1c4effd640 1 -- 172.21.15.55:0/953272116 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f1c500520c0 con 0x7f1c601473a0 2024-03-25T18:28:57.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.615+0000 7f1c4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:57.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.615+0000 7f1c4effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:57.617 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:28:57.617 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 10 librados: watch_flush enter 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 10 librados: watch_flush exit 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 10 client.4266.objecter successfully canceled tick 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 >> v1:172.21.15.55:6800/286536064 conn(0x7f1c4006dfb0 legacy=0x7f1c40070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 10 monclient: shutdown 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 >> v1:172.21.15.55:6789/0 conn(0x7f1c601473a0 legacy=0x7f1c6013ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 shutdown_connections 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 >> 172.21.15.55:0/953272116 conn(0x7f1c600b7ea0 msgr2=0x7f1c601416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 shutdown_connections 2024-03-25T18:28:57.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 1 -- 172.21.15.55:0/953272116 wait complete. 2024-03-25T18:28:57.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:28:57.617+0000 7f1c64d2b640 1 librados: shutdown 2024-03-25T18:28:57.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:04.436 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:04.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.556+0000 7f8503158640 10 monclient: get_monmap_and_config 2024-03-25T18:29:04.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.556+0000 7f8503158640 10 monclient: build_initial_monmap 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.556+0000 7f8503158640 10 monclient: monmap: 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:29:04.557428+0000 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:29:04.557428+0000 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:29:04.558 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.557+0000 7f8503158640 1 Processor -- start 2024-03-25T18:29:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.557+0000 7f8503158640 1 -- start start 2024-03-25T18:29:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.557+0000 7f8503158640 10 monclient: init 2024-03-25T18:29:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:04.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 10 monclient(hunting): picked mon.noname-b con 0x7f84fc0a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 10 monclient(hunting): picked mon.noname-a con 0x7f84fc13f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 10 monclient(hunting): picked mon.noname-c con 0x7f84fc1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc142520 con 0x7f84fc13f710 2024-03-25T18:29:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc0b28a0 con 0x7f84fc1430c0 2024-03-25T18:29:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc0b2a30 con 0x7f84fc0a98f0 2024-03-25T18:29:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:04.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.558+0000 7f8503158640 10 monclient(hunting): authenticate will time out at 7348.815547s 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.567+0000 7f8500ef4640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f84fc0a98f0 0x7f84fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:59630/0 (socket says 172.21.15.55:59630) 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.567+0000 7f8500ef4640 1 -- 172.21.15.55:0/2754059959 learned_addr learned my addr 172.21.15.55:0/2754059959 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:04.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.580+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940042236 0 0) 0x7f84fc0b2a30 con 0x7f84fc0a98f0 2024-03-25T18:29:04.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.580+0000 7f84fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.580+0000 7f84fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.580+0000 7f84fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.580+0000 7f84fb7fe640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:29:04.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.580+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84ec003520 con 0x7f84fc0a98f0 2024-03-25T18:29:04.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.589+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658784768 0 0) 0x7f84fc0b28a0 con 0x7f84fc1430c0 2024-03-25T18:29:04.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.589+0000 7f84fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.589+0000 7f84fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.589+0000 7f84fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.589+0000 7f84fb7fe640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:29:04.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.589+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84fc0b2a30 con 0x7f84fc1430c0 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3663907024 0 0) 0x7f84ec003520 con 0x7f84fc0a98f0 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 >> v1:172.21.15.133:6789/0 conn(0x7f84fc1430c0 legacy=0x7f84fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 >> v1:172.21.15.55:6789/0 conn(0x7f84fc13f710 legacy=0x7f84fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 1 monclient: found mon.noname-b 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc0b44c0 con 0x7f84fc0a98f0 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f84f0001690 con 0x7f84fc0a98f0 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f84fb7fe640 10 monclient: dump: 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.593 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.593 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.593 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:04.593 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:04.593 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:04.593 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:04.593 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:04.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f8503158640 5 monclient: authenticate success, global_id 4265 2024-03-25T18:29:04.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.590+0000 7f8503158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:04.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84f00017f0 con 0x7f84fc0a98f0 2024-03-25T18:29:04.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f84fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:04.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f8503158640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:04.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f8503158640 4 set_mon_vals no callback set 2024-03-25T18:29:04.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f84fb7fe640 1 -- 172.21.15.55:0/2754059959 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f84f0001c00 con 0x7f84fc0a98f0 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f84fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f84fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f84fb7fe640 10 monclient: dump: 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:04.597 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f8503158640 10 monclient: shutdown 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f8503158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.594+0000 7f8503158640 1 -- 172.21.15.55:0/2754059959 >> v1:172.21.15.55:6790/0 conn(0x7f84fc0a98f0 legacy=0x7f84fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.595+0000 7f8503158640 1 -- 172.21.15.55:0/2754059959 shutdown_connections 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.595+0000 7f8503158640 1 -- 172.21.15.55:0/2754059959 >> 172.21.15.55:0/2754059959 conn(0x7f84fc0b7ea0 msgr2=0x7f84fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.595+0000 7f8503158640 1 -- 172.21.15.55:0/2754059959 shutdown_connections 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.595+0000 7f8503158640 1 -- 172.21.15.55:0/2754059959 wait complete. 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.595+0000 7f8503158640 10 monclient: build_initial_monmap 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.595+0000 7f8503158640 10 monclient: monmap: 2024-03-25T18:29:04.598 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.596+0000 7f8503158640 1 librados: starting msgr at 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.596+0000 7f8503158640 1 librados: starting objecter 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.596+0000 7f8503158640 1 Processor -- start 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.596+0000 7f8503158640 1 -- start start 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.596+0000 7f8503158640 1 librados: setting wanted keys 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.596+0000 7f8503158640 1 librados: calling monclient init 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.596+0000 7f8503158640 10 monclient: init 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 10 monclient(hunting): picked mon.noname-b con 0x7f84fc0a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 10 monclient(hunting): picked mon.noname-a con 0x7f84fc13f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 10 monclient(hunting): picked mon.noname-c con 0x7f84fc1430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc1d51f0 con 0x7f84fc13f710 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc1dac30 con 0x7f84fc0a98f0 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc0ab9e0 con 0x7f84fc1430c0 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.597+0000 7f8503158640 10 monclient(hunting): authenticate will time out at 7348.854829s 2024-03-25T18:29:04.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.606+0000 7f8500ef4640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f84fc0a98f0 0x7f84fc1382c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:38562/0 (socket says 172.21.15.55:38562) 2024-03-25T18:29:04.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.606+0000 7f8500ef4640 1 -- 172.21.15.55:0/806160522 learned_addr learned my addr 172.21.15.55:0/806160522 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:04.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.609+0000 7f84f97fa640 10 client.?.objecter ms_handle_connect 0x7f84fc13f710 2024-03-25T18:29:04.610 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.609+0000 7f84f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:04.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.610+0000 7f84f97fa640 10 client.?.objecter ms_handle_connect 0x7f84fc0a98f0 2024-03-25T18:29:04.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.610+0000 7f84f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:04.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.611+0000 7f84f97fa640 10 client.?.objecter ms_handle_connect 0x7f84fc1430c0 2024-03-25T18:29:04.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.611+0000 7f84f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:04.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.616+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273666953 0 0) 0x7f84fc1d51f0 con 0x7f84fc13f710 2024-03-25T18:29:04.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.616+0000 7f84f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.616+0000 7f84f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.616+0000 7f84f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.616+0000 7f84f97fa640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:29:04.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.616+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e0004eb0 con 0x7f84fc13f710 2024-03-25T18:29:04.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116527761 0 0) 0x7f84fc0ab9e0 con 0x7f84fc1430c0 2024-03-25T18:29:04.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:29:04.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84fc1d51f0 con 0x7f84fc1430c0 2024-03-25T18:29:04.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393676272 0 0) 0x7f84fc1dac30 con 0x7f84fc0a98f0 2024-03-25T18:29:04.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:29:04.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.619+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84fc0ab9e0 con 0x7f84fc0a98f0 2024-03-25T18:29:04.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.628+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 814445483 0 0) 0x7f84e0004eb0 con 0x7f84fc13f710 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84fc1dac30 con 0x7f84fc13f710 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f84e8001930 con 0x7f84fc13f710 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 10 monclient(hunting): dump: 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:04.630 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:04.631 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:04.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:04.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1147536865 0 0) 0x7f84fc1d51f0 con 0x7f84fc1430c0 2024-03-25T18:29:04.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84e0004eb0 con 0x7f84fc1430c0 2024-03-25T18:29:04.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f84f4001b90 con 0x7f84fc1430c0 2024-03-25T18:29:04.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.629+0000 7f84f97fa640 10 monclient(hunting): dump: 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:04.632 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 504118239 0 0) 0x7f84fc1dac30 con 0x7f84fc13f710 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 >> v1:172.21.15.55:6790/0 conn(0x7f84fc1430c0 legacy=0x7f84fc1e1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 >> v1:172.21.15.133:6789/0 conn(0x7f84fc0a98f0 legacy=0x7f84fc1382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 1 monclient: found mon.a 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc1389d0 con 0x7f84fc13f710 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 10 monclient: _finish_auth 0 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f84f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f8503158640 5 monclient: authenticate success, global_id 4272 2024-03-25T18:29:04.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f8503158640 10 monclient: _renew_subs 2024-03-25T18:29:04.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f8503158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:04.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.633+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84fc1db860 con 0x7f84fc13f710 2024-03-25T18:29:04.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.634+0000 7f8503158640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:04.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.634+0000 7f8503158640 10 monclient: _renew_subs 2024-03-25T18:29:04.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.634+0000 7f8503158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:04.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.634+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84fc138e20 con 0x7f84fc13f710 2024-03-25T18:29:04.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.634+0000 7f8503158640 1 librados: init done 2024-03-25T18:29:04.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.635+0000 7f8503158640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:04.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.635+0000 7f8503158640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:04.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.635+0000 7f8503158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:04.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.635+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84fc1dadc0 con 0x7f84fc13f710 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.664+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84e80035f0 con 0x7f84fc13f710 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.664+0000 7f84f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.664+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f84e8003980 con 0x7f84fc13f710 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.664+0000 7f84f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.665+0000 7f84f97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.665+0000 7f84f97fa640 10 monclient: dump: 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:04.666 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.667 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:04.667 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:04.667 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:04.667 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:04.667 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:04.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.664+0000 7f84faffd640 4 set_mon_vals no callback set 2024-03-25T18:29:04.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.665+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f84e8003ba0 con 0x7f84fc13f710 2024-03-25T18:29:04.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.666+0000 7f8500ef4640 10 monclient: get_auth_request con 0x7f84e006df70 auth_method 0 2024-03-25T18:29:04.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.666+0000 7f84f97fa640 10 client.4272.objecter ms_handle_connect 0x7f84e006df70 2024-03-25T18:29:04.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.669+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f84e8053650 con 0x7f84fc13f710 2024-03-25T18:29:04.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.669+0000 7f84f97fa640 10 client.4272.objecter ms_dispatch 0x7f84fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:04.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.669+0000 7f84f97fa640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:04.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.669+0000 7f84f97fa640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:04.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.669+0000 7f84f97fa640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-25T18:29:04.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.987+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84e804f160 con 0x7f84fc13f710 2024-03-25T18:29:04.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.987+0000 7f84f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:04.988 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:04.987+0000 7f84f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:05.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.241+0000 7f8503158640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:05.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.241+0000 7f8503158640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:05.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.241+0000 7f8503158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:05.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.241+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84fc0627b0 con 0x7f84fc13f710 2024-03-25T18:29:05.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.303+0000 7f84f97fa640 1 -- 172.21.15.55:0/806160522 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f84e8051090 con 0x7f84fc13f710 2024-03-25T18:29:05.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.303+0000 7f84f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:05.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.303+0000 7f84f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:05.304 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:29:05.305 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.306+0000 7f8503158640 10 librados: watch_flush enter 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.306+0000 7f8503158640 10 librados: watch_flush exit 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.306+0000 7f8503158640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.306+0000 7f8503158640 10 client.4272.objecter successfully canceled tick 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.306+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 >> v1:172.21.15.55:6800/286536064 conn(0x7f84e006df70 legacy=0x7f84e0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.306+0000 7f8503158640 10 monclient: shutdown 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.306+0000 7f8503158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.306+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 >> v1:172.21.15.55:6789/0 conn(0x7f84fc13f710 legacy=0x7f84fc1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.307+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 shutdown_connections 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.307+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 >> 172.21.15.55:0/806160522 conn(0x7f84fc0b7ea0 msgr2=0x7f84fc13ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:05.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.307+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 shutdown_connections 2024-03-25T18:29:05.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.307+0000 7f8503158640 1 -- 172.21.15.55:0/806160522 wait complete. 2024-03-25T18:29:05.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:05.307+0000 7f8503158640 1 librados: shutdown 2024-03-25T18:29:05.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:09.226 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:01:19.396 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.323 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:01:19.493 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.478 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:19.615 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.480 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:01:19.650 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.545 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:19.682 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.553 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:01:19.723 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.583 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:01:19.753 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.770 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:19.907 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.788 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:01:19.958 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:12.124 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:12.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.241+0000 7f15f4a0f640 10 monclient: get_monmap_and_config 2024-03-25T18:29:12.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.241+0000 7f15f4a0f640 10 monclient: build_initial_monmap 2024-03-25T18:29:12.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.241+0000 7f15f4a0f640 10 monclient: monmap: 2024-03-25T18:29:12.242 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:12.243 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:12.243 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:29:12.242520+0000 2024-03-25T18:29:12.243 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:29:12.242520+0000 2024-03-25T18:29:12.243 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:12.243 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:12.243 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:12.243 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:29:12.243 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:29:12.243 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:12.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.242+0000 7f15f4a0f640 1 Processor -- start 2024-03-25T18:29:12.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.242+0000 7f15f4a0f640 1 -- start start 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.242+0000 7f15f4a0f640 10 monclient: init 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.242+0000 7f15f4a0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.242+0000 7f15f4a0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.242+0000 7f15f4a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f15f00a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.243+0000 7f15f4a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f15f013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.243+0000 7f15f4a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f15f01430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.243+0000 7f15f4a0f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f0142520 con 0x7f15f01430c0 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.243+0000 7f15f4a0f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f00b28a0 con 0x7f15f00a98f0 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.243+0000 7f15f4a0f640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f00b2a30 con 0x7f15f013f710 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.243+0000 7f15f4a0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.243+0000 7f15f4a0f640 10 monclient(hunting): authenticate will time out at 7356.500441s 2024-03-25T18:29:12.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.252+0000 7f15e5d9b640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f15f013f710 0x7f15f01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:54282/0 (socket says 172.21.15.55:54282) 2024-03-25T18:29:12.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.252+0000 7f15e5d9b640 1 -- 172.21.15.55:0/2380378994 learned_addr learned my addr 172.21.15.55:0/2380378994 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.262+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941682318 0 0) 0x7f15f00b2a30 con 0x7f15f013f710 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.262+0000 7f15edd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.262+0000 7f15edd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.262+0000 7f15edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.262+0000 7f15edd9b640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.262+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15dc003520 con 0x7f15f013f710 2024-03-25T18:29:12.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.268+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730890623 0 0) 0x7f15f0142520 con 0x7f15f01430c0 2024-03-25T18:29:12.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.268+0000 7f15edd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.268+0000 7f15edd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.268+0000 7f15edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.268+0000 7f15edd9b640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:29:12.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.268+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f00b2a30 con 0x7f15f01430c0 2024-03-25T18:29:12.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.273+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577039033 0 0) 0x7f15f00b28a0 con 0x7f15f00a98f0 2024-03-25T18:29:12.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.273+0000 7f15edd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.273+0000 7f15edd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.273+0000 7f15edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.273+0000 7f15edd9b640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:29:12.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.273+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f0142520 con 0x7f15f00a98f0 2024-03-25T18:29:12.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2368050097 0 0) 0x7f15dc003520 con 0x7f15f013f710 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 >> v1:172.21.15.133:6789/0 conn(0x7f15f00a98f0 legacy=0x7f15f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 >> v1:172.21.15.55:6789/0 conn(0x7f15f01430c0 legacy=0x7f15f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 1 monclient: found mon.noname-b 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f00b44c0 con 0x7f15f013f710 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 10 monclient: _finish_auth 0 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:12.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f15d8001690 con 0x7f15f013f710 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15f4a0f640 5 monclient: authenticate success, global_id 4271 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15f4a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15edd9b640 10 monclient: dump: 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:12.278 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:12.279 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:12.279 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:12.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.275+0000 7f15f4a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:12.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.279+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15d80017f0 con 0x7f15f013f710 2024-03-25T18:29:12.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.279+0000 7f15edd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:12.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.279+0000 7f15edd9b640 1 -- 172.21.15.55:0/2380378994 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f15d8001c00 con 0x7f15f013f710 2024-03-25T18:29:12.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.279+0000 7f15edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.279+0000 7f15edd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.279+0000 7f15edd9b640 10 monclient: dump: 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.279+0000 7f15f4a0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.279+0000 7f15f4a0f640 4 set_mon_vals no callback set 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.280+0000 7f15f4a0f640 10 monclient: shutdown 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.280+0000 7f15f4a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.280+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2380378994 >> v1:172.21.15.55:6790/0 conn(0x7f15f013f710 legacy=0x7f15f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.280+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2380378994 shutdown_connections 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.280+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2380378994 >> 172.21.15.55:0/2380378994 conn(0x7f15f00b7ea0 msgr2=0x7f15f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.280+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2380378994 shutdown_connections 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.280+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2380378994 wait complete. 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.281+0000 7f15f4a0f640 10 monclient: build_initial_monmap 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.281+0000 7f15f4a0f640 10 monclient: monmap: 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:29:12.284 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.281+0000 7f15f4a0f640 1 librados: starting msgr at 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.281+0000 7f15f4a0f640 1 librados: starting objecter 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 1 Processor -- start 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 1 -- start start 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 1 librados: setting wanted keys 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 1 librados: calling monclient init 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 10 monclient: init 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:12.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:12.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f15f00a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:12.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f15f013f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:12.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f15f01430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:12.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f01370a0 con 0x7f15f00a98f0 2024-03-25T18:29:12.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f0135090 con 0x7f15f013f710 2024-03-25T18:29:12.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f0135220 con 0x7f15f01430c0 2024-03-25T18:29:12.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:12.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.282+0000 7f15f4a0f640 10 monclient(hunting): authenticate will time out at 7356.540187s 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.289+0000 7f15e5d9b640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f15f013f710 0x7f15f01d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:55522/0 (socket says 172.21.15.55:55522) 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.289+0000 7f15e5d9b640 1 -- 172.21.15.55:0/2569353421 learned_addr learned my addr 172.21.15.55:0/2569353421 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:12.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.293+0000 7f15e77fe640 10 client.?.objecter ms_handle_connect 0x7f15f013f710 2024-03-25T18:29:12.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.293+0000 7f15e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:12.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.304+0000 7f15e77fe640 10 client.?.objecter ms_handle_connect 0x7f15f01430c0 2024-03-25T18:29:12.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.304+0000 7f15e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:12.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.307+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602107470 0 0) 0x7f15f0135090 con 0x7f15f013f710 2024-03-25T18:29:12.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.307+0000 7f15e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.307+0000 7f15e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.307+0000 7f15e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.307+0000 7f15e77fe640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:29:12.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.307+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d0004eb0 con 0x7f15f013f710 2024-03-25T18:29:12.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.309+0000 7f15e77fe640 10 client.?.objecter ms_handle_connect 0x7f15f00a98f0 2024-03-25T18:29:12.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.309+0000 7f15e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:12.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.314+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468969193 0 0) 0x7f15f0135220 con 0x7f15f01430c0 2024-03-25T18:29:12.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.314+0000 7f15e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.314+0000 7f15e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.314+0000 7f15e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.314+0000 7f15e77fe640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:29:12.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.314+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f0135090 con 0x7f15f01430c0 2024-03-25T18:29:12.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.320+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242451610 0 0) 0x7f15f01370a0 con 0x7f15f00a98f0 2024-03-25T18:29:12.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.320+0000 7f15e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.320+0000 7f15e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.320+0000 7f15e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.320+0000 7f15e77fe640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:29:12.321 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.320+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f0135220 con 0x7f15f00a98f0 2024-03-25T18:29:12.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.332+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2089310873 0 0) 0x7f15f0135090 con 0x7f15f01430c0 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.332+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f01370a0 con 0x7f15f01430c0 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.332+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f15e8001b50 con 0x7f15f01430c0 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.332+0000 7f15e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.332+0000 7f15e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.332+0000 7f15e77fe640 10 monclient(hunting): dump: 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:12.333 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:12.334 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:12.334 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:12.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.332+0000 7f15e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1897942752 0 0) 0x7f15d0004eb0 con 0x7f15f013f710 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f0135090 con 0x7f15f013f710 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f15d8001bc0 con 0x7f15f013f710 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 10 monclient(hunting): dump: 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.343 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3323094548 0 0) 0x7f15f01370a0 con 0x7f15f01430c0 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 >> v1:172.21.15.133:6789/0 conn(0x7f15f013f710 legacy=0x7f15f01d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 >> v1:172.21.15.55:6789/0 conn(0x7f15f00a98f0 legacy=0x7f15f01dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 1 monclient: found mon.c 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f01d5000 con 0x7f15f01430c0 2024-03-25T18:29:12.344 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15f4a0f640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15f4a0f640 10 monclient: _renew_subs 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15f4a0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f01d52b0 con 0x7f15f01430c0 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15f4a0f640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15f4a0f640 10 monclient: _renew_subs 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15f4a0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.341+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f0135910 con 0x7f15f01430c0 2024-03-25T18:29:12.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.342+0000 7f15f4a0f640 1 librados: init done 2024-03-25T18:29:12.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.343+0000 7f15f4a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:12.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.343+0000 7f15f4a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:12.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.343+0000 7f15f4a0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:12.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.343+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f00627b0 con 0x7f15f01430c0 2024-03-25T18:29:12.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.655+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15e8002060 con 0x7f15f01430c0 2024-03-25T18:29:12.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.655+0000 7f15e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:12.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.655+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f15e8003b10 con 0x7f15f01430c0 2024-03-25T18:29:12.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.655+0000 7f15e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.655+0000 7f15e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.655+0000 7f15e77fe640 10 monclient: dump: 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:12.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.655+0000 7f15ed59a640 4 set_mon_vals no callback set 2024-03-25T18:29:12.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.656+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f15e8019b20 con 0x7f15f01430c0 2024-03-25T18:29:12.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.657+0000 7f15ee59c640 10 monclient: get_auth_request con 0x7f15d006de80 auth_method 0 2024-03-25T18:29:12.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.657+0000 7f15e77fe640 10 client.4274.objecter ms_handle_connect 0x7f15d006de80 2024-03-25T18:29:12.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.662+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f15e80538d0 con 0x7f15f01430c0 2024-03-25T18:29:12.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.662+0000 7f15e77fe640 10 client.4274.objecter ms_dispatch 0x7f15f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:12.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.662+0000 7f15e77fe640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:12.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.662+0000 7f15e77fe640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:12.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.662+0000 7f15e77fe640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-25T18:29:12.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.662+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15e804f210 con 0x7f15f01430c0 2024-03-25T18:29:12.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.662+0000 7f15e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:12.664 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.662+0000 7f15e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:12.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.910+0000 7f15f4a0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:12.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.910+0000 7f15f4a0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:12.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.910+0000 7f15f4a0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:12.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.910+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15f014f240 con 0x7f15f01430c0 2024-03-25T18:29:12.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.933+0000 7f15e77fe640 1 -- 172.21.15.55:0/2569353421 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f15e8051310 con 0x7f15f01430c0 2024-03-25T18:29:12.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.933+0000 7f15e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:12.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.933+0000 7f15e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:12.935 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:29:12.935 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:12.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 10 librados: watch_flush enter 2024-03-25T18:29:12.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 10 librados: watch_flush exit 2024-03-25T18:29:12.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-25T18:29:12.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 10 client.4274.objecter successfully canceled tick 2024-03-25T18:29:12.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 >> v1:172.21.15.55:6800/286536064 conn(0x7f15d006de80 legacy=0x7f15d0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 10 monclient: shutdown 2024-03-25T18:29:12.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:12.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 >> v1:172.21.15.55:6790/0 conn(0x7f15f01430c0 legacy=0x7f15f01d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 shutdown_connections 2024-03-25T18:29:12.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 >> 172.21.15.55:0/2569353421 conn(0x7f15f00b7ea0 msgr2=0x7f15f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:12.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 shutdown_connections 2024-03-25T18:29:12.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 1 -- 172.21.15.55:0/2569353421 wait complete. 2024-03-25T18:29:12.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:12.936+0000 7f15f4a0f640 1 librados: shutdown 2024-03-25T18:29:12.949 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:19.756 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.873+0000 7fe69093a640 10 monclient: get_monmap_and_config 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.873+0000 7fe69093a640 10 monclient: build_initial_monmap 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.873+0000 7fe69093a640 10 monclient: monmap: 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:29:19.873992+0000 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:29:19.873992+0000 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:19.874 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:29:19.875 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:29:19.875 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.873+0000 7fe69093a640 1 Processor -- start 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.873+0000 7fe69093a640 1 -- start start 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.873+0000 7fe69093a640 10 monclient: init 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 10 monclient(hunting): picked mon.noname-a con 0x7fe68c0b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 10 monclient(hunting): picked mon.noname-b con 0x7fe68c0b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 10 monclient(hunting): picked mon.noname-c con 0x7fe68c14be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c0af940 con 0x7fe68c0b4d60 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c0b0a70 con 0x7fe68c14be90 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c0b0d50 con 0x7fe68c0b1730 2024-03-25T18:29:19.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:19.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.874+0000 7fe69093a640 10 monclient(hunting): authenticate will time out at 7364.132221s 2024-03-25T18:29:19.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.893+0000 7fe68a59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fe68c0b4d60 0x7fe68c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:36432/0 (socket says 172.21.15.55:36432) 2024-03-25T18:29:19.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.894+0000 7fe68a59c640 1 -- 172.21.15.55:0/3635207757 learned_addr learned my addr 172.21.15.55:0/3635207757 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.904+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928502033 0 0) 0x7fe68c0b0a70 con 0x7fe68c14be90 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.904+0000 7fe68959a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.904+0000 7fe68959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.904+0000 7fe68959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.904+0000 7fe68959a640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.904+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe674003520 con 0x7fe68c14be90 2024-03-25T18:29:19.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.922+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761766494 0 0) 0x7fe68c0af940 con 0x7fe68c0b4d60 2024-03-25T18:29:19.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.922+0000 7fe68959a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.922+0000 7fe68959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.922+0000 7fe68959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.922+0000 7fe68959a640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:29:19.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.922+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe68c0b0a70 con 0x7fe68c0b4d60 2024-03-25T18:29:19.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2601460567 0 0) 0x7fe674003520 con 0x7fe68c14be90 2024-03-25T18:29:19.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 >> v1:172.21.15.55:6790/0 conn(0x7fe68c0b1730 legacy=0x7fe68c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 >> v1:172.21.15.55:6789/0 conn(0x7fe68c0b4d60 legacy=0x7fe68c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 10 monclient: _finish_hunting 0 2024-03-25T18:29:19.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 1 monclient: found mon.noname-c 2024-03-25T18:29:19.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c0b1030 con 0x7fe68c14be90 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 10 monclient: _finish_auth 0 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 10 monclient: _check_auth_tickets 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fe684001a10 con 0x7fe68c14be90 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe69093a640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe69093a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe68959a640 10 monclient: dump: 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:19.928 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:19.929 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.929 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:19.929 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:19.929 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:19.929 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:19.929 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:19.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.926+0000 7fe69093a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:19.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.951+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe684001b70 con 0x7fe68c14be90 2024-03-25T18:29:19.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.951+0000 7fe68959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:19.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.951+0000 7fe68959a640 1 -- 172.21.15.55:0/3635207757 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fe684001f80 con 0x7fe68c14be90 2024-03-25T18:29:19.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.951+0000 7fe68959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.951+0000 7fe68959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:19.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.951+0000 7fe68959a640 10 monclient: dump: 2024-03-25T18:29:19.952 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:19.952 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.951+0000 7fe69093a640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:19.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.951+0000 7fe69093a640 4 set_mon_vals no callback set 2024-03-25T18:29:19.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.952+0000 7fe69093a640 10 monclient: shutdown 2024-03-25T18:29:19.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.952+0000 7fe69093a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:19.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.952+0000 7fe69093a640 1 -- 172.21.15.55:0/3635207757 >> v1:172.21.15.133:6789/0 conn(0x7fe68c14be90 legacy=0x7fe68c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.952+0000 7fe69093a640 1 -- 172.21.15.55:0/3635207757 shutdown_connections 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.952+0000 7fe69093a640 1 -- 172.21.15.55:0/3635207757 >> 172.21.15.55:0/3635207757 conn(0x7fe68c0617a0 msgr2=0x7fe68c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.952+0000 7fe69093a640 1 -- 172.21.15.55:0/3635207757 shutdown_connections 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.952+0000 7fe69093a640 1 -- 172.21.15.55:0/3635207757 wait complete. 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.953+0000 7fe69093a640 10 monclient: build_initial_monmap 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.953+0000 7fe69093a640 10 monclient: monmap: 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:19.955 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:29:19.956 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:29:19.956 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:19.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 1 librados: starting msgr at 2024-03-25T18:29:19.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 1 librados: starting objecter 2024-03-25T18:29:19.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 1 Processor -- start 2024-03-25T18:29:19.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 1 -- start start 2024-03-25T18:29:19.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 1 librados: setting wanted keys 2024-03-25T18:29:19.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 1 librados: calling monclient init 2024-03-25T18:29:19.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 10 monclient: init 2024-03-25T18:29:19.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:19.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:19.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.954+0000 7fe69093a640 10 monclient(hunting): picked mon.noname-a con 0x7fe68c0b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:19.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.955+0000 7fe69093a640 10 monclient(hunting): picked mon.noname-c con 0x7fe68c0b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:19.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.955+0000 7fe69093a640 10 monclient(hunting): picked mon.noname-b con 0x7fe68c14be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:19.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.955+0000 7fe69093a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c1363f0 con 0x7fe68c0b1730 2024-03-25T18:29:19.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.955+0000 7fe69093a640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c1374e0 con 0x7fe68c14be90 2024-03-25T18:29:19.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.955+0000 7fe69093a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c138140 con 0x7fe68c0b4d60 2024-03-25T18:29:19.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.955+0000 7fe69093a640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:19.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.955+0000 7fe69093a640 10 monclient(hunting): authenticate will time out at 7364.212445s 2024-03-25T18:29:19.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.967+0000 7fe68ad9d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fe68c14be90 0x7fe68c143c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:48518/0 (socket says 172.21.15.55:48518) 2024-03-25T18:29:19.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.967+0000 7fe68ad9d640 1 -- 172.21.15.55:0/2972512299 learned_addr learned my addr 172.21.15.55:0/2972512299 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:19.974 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.973+0000 7fe67affd640 10 client.?.objecter ms_handle_connect 0x7fe68c14be90 2024-03-25T18:29:19.974 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.973+0000 7fe67affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:19.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.983+0000 7fe67affd640 10 client.?.objecter ms_handle_connect 0x7fe68c0b1730 2024-03-25T18:29:19.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.983+0000 7fe67affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:19.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.984+0000 7fe67affd640 10 client.?.objecter ms_handle_connect 0x7fe68c0b4d60 2024-03-25T18:29:19.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.984+0000 7fe67affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:19.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.992+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275601336 0 0) 0x7fe68c1374e0 con 0x7fe68c14be90 2024-03-25T18:29:19.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.992+0000 7fe67affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.992+0000 7fe67affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.992+0000 7fe67affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.992+0000 7fe67affd640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:29:19.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:19.992+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe66c004eb0 con 0x7fe68c14be90 2024-03-25T18:29:20.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.000+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125857843 0 0) 0x7fe68c138140 con 0x7fe68c0b4d60 2024-03-25T18:29:20.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.000+0000 7fe67affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:20.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.000+0000 7fe67affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:20.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.000+0000 7fe67affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:20.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.000+0000 7fe67affd640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:29:20.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.000+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe68c1374e0 con 0x7fe68c0b4d60 2024-03-25T18:29:20.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.003+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722980700 0 0) 0x7fe68c1363f0 con 0x7fe68c0b1730 2024-03-25T18:29:20.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.003+0000 7fe67affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:20.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.003+0000 7fe67affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:20.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.003+0000 7fe67affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:20.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.003+0000 7fe67affd640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:29:20.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.003+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe68c138140 con 0x7fe68c0b1730 2024-03-25T18:29:20.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.009+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 164933964 0 0) 0x7fe66c004eb0 con 0x7fe68c14be90 2024-03-25T18:29:20.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.009+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe68c1363f0 con 0x7fe68c14be90 2024-03-25T18:29:20.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.009+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fe684001f40 con 0x7fe68c14be90 2024-03-25T18:29:20.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.009+0000 7fe67affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:20.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.009+0000 7fe67affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:20.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.009+0000 7fe67affd640 10 monclient(hunting): dump: 2024-03-25T18:29:20.010 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:20.010 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:20.011 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:20.011 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:20.011 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:20.011 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:20.011 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:20.011 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:20.011 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:20.011 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:20.011 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.009+0000 7fe67affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:20.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2839108787 0 0) 0x7fe68c1363f0 con 0x7fe68c14be90 2024-03-25T18:29:20.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 >> v1:172.21.15.55:6790/0 conn(0x7fe68c0b4d60 legacy=0x7fe68c12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 >> v1:172.21.15.55:6789/0 conn(0x7fe68c0b1730 legacy=0x7fe68c136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 1 monclient: found mon.b 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c139090 con 0x7fe68c14be90 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 10 monclient: _finish_auth 0 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe67affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:20.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe69093a640 5 monclient: authenticate success, global_id 4264 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe69093a640 10 monclient: _renew_subs 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe69093a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.021+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68c136030 con 0x7fe68c14be90 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.022+0000 7fe69093a640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.022+0000 7fe69093a640 10 monclient: _renew_subs 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.022+0000 7fe69093a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.022+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68c130650 con 0x7fe68c14be90 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.022+0000 7fe69093a640 1 librados: init done 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.022+0000 7fe69093a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.022+0000 7fe69093a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.023+0000 7fe69093a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:20.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.023+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68c0ae6c0 con 0x7fe68c14be90 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.296+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe684001510 con 0x7fe68c14be90 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.296+0000 7fe67affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.296+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fe684003dc0 con 0x7fe68c14be90 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.296+0000 7fe67affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.296+0000 7fe67affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.296+0000 7fe67affd640 10 monclient: dump: 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:20.298 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:20.299 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:20.299 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:20.299 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:20.299 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:20.299 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:20.299 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:20.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.296+0000 7fe688d99640 4 set_mon_vals no callback set 2024-03-25T18:29:20.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.297+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fe684003fe0 con 0x7fe68c14be90 2024-03-25T18:29:20.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.298+0000 7fe68a59c640 10 monclient: get_auth_request con 0x7fe66c06ddf0 auth_method 0 2024-03-25T18:29:20.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.298+0000 7fe67affd640 10 client.4264.objecter ms_handle_connect 0x7fe66c06ddf0 2024-03-25T18:29:20.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.304+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fe684054a90 con 0x7fe68c14be90 2024-03-25T18:29:20.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.304+0000 7fe67affd640 10 client.4264.objecter ms_dispatch 0x7fe68c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:20.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.304+0000 7fe67affd640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:20.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.304+0000 7fe67affd640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:20.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.304+0000 7fe67affd640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T18:29:20.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.304+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6840505a0 con 0x7fe68c14be90 2024-03-25T18:29:20.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.305+0000 7fe67affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:20.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.305+0000 7fe67affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:20.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.569+0000 7fe69093a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:20.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.569+0000 7fe69093a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:20.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.569+0000 7fe69093a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:20.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.569+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe68c0627b0 con 0x7fe68c14be90 2024-03-25T18:29:20.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.629+0000 7fe67affd640 1 -- 172.21.15.55:0/2972512299 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fe6840524d0 con 0x7fe68c14be90 2024-03-25T18:29:20.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.629+0000 7fe67affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:20.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.629+0000 7fe67affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:20.631 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:29:20.631 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:20.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.631+0000 7fe69093a640 10 librados: watch_flush enter 2024-03-25T18:29:20.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.631+0000 7fe69093a640 10 librados: watch_flush exit 2024-03-25T18:29:20.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.631+0000 7fe69093a640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T18:29:20.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.631+0000 7fe69093a640 10 client.4264.objecter successfully canceled tick 2024-03-25T18:29:20.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.631+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 >> v1:172.21.15.55:6800/286536064 conn(0x7fe66c06ddf0 legacy=0x7fe66c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:20.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.631+0000 7fe69093a640 10 monclient: shutdown 2024-03-25T18:29:20.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.631+0000 7fe69093a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:20.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.631+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 >> v1:172.21.15.133:6789/0 conn(0x7fe68c14be90 legacy=0x7fe68c143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:20.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.632+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 shutdown_connections 2024-03-25T18:29:20.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.632+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 >> 172.21.15.55:0/2972512299 conn(0x7fe68c0617a0 msgr2=0x7fe68c13c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:20.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.632+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 shutdown_connections 2024-03-25T18:29:20.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.632+0000 7fe69093a640 1 -- 172.21.15.55:0/2972512299 wait complete. 2024-03-25T18:29:20.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:20.632+0000 7fe69093a640 1 librados: shutdown 2024-03-25T18:29:20.644 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:27.452 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:27.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.566+0000 7f716bc55640 10 monclient: get_monmap_and_config 2024-03-25T18:29:27.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.566+0000 7f716bc55640 10 monclient: build_initial_monmap 2024-03-25T18:29:27.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.566+0000 7f716bc55640 10 monclient: monmap: 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:29:27.568580+0000 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:29:27.568580+0000 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.567+0000 7f716bc55640 1 Processor -- start 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.567+0000 7f716bc55640 1 -- start start 2024-03-25T18:29:27.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.567+0000 7f716bc55640 10 monclient: init 2024-03-25T18:29:27.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 10 monclient(hunting): picked mon.noname-b con 0x7f71640b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 10 monclient(hunting): picked mon.noname-c con 0x7f71640b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 10 monclient(hunting): picked mon.noname-a con 0x7f716414be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71640af940 con 0x7f716414be90 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71640b0a70 con 0x7f71640b1730 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71640b0d50 con 0x7f71640b4d60 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.568+0000 7f716bc55640 10 monclient(hunting): authenticate will time out at 7371.827061s 2024-03-25T18:29:27.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.582+0000 7f71691f0640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f71640b1730 0x7f71640b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:53310/0 (socket says 172.21.15.55:53310) 2024-03-25T18:29:27.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.582+0000 7f71691f0640 1 -- 172.21.15.55:0/2770000373 learned_addr learned my addr 172.21.15.55:0/2770000373 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:27.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.593+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468158482 0 0) 0x7f71640b0a70 con 0x7f71640b1730 2024-03-25T18:29:27.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.593+0000 7f71689ef640 10 monclient(hunting): _negotiate 2024-03-25T18:29:27.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.593+0000 7f71689ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:27.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.593+0000 7f71689ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:27.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.593+0000 7f71689ef640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:29:27.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.593+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7150003520 con 0x7f71640b1730 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.596+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679560589 0 0) 0x7f71640af940 con 0x7f716414be90 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.596+0000 7f71689ef640 10 monclient(hunting): _negotiate 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.596+0000 7f71689ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.596+0000 7f71689ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.596+0000 7f71689ef640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:29:27.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.596+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71640b0a70 con 0x7f716414be90 2024-03-25T18:29:27.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1508390090 0 0) 0x7f7150003520 con 0x7f71640b1730 2024-03-25T18:29:27.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 >> v1:172.21.15.55:6790/0 conn(0x7f71640b4d60 legacy=0x7f716413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 >> v1:172.21.15.55:6789/0 conn(0x7f716414be90 legacy=0x7f716414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 10 monclient: _finish_hunting 0 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 1 monclient: found mon.noname-c 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71640b1030 con 0x7f71640b1730 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 10 monclient: _finish_auth 0 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 10 monclient: _check_auth_tickets 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7154001690 con 0x7f71640b1730 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f71689ef640 10 monclient: dump: 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f716bc55640 5 monclient: authenticate success, global_id 4267 2024-03-25T18:29:27.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.600+0000 7f716bc55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:27.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71540017f0 con 0x7f71640b1730 2024-03-25T18:29:27.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f71689ef640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f71689ef640 1 -- 172.21.15.55:0/2770000373 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7154001c00 con 0x7f71640b1730 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f716bc55640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f716bc55640 4 set_mon_vals no callback set 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f71689ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f71689ef640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f71689ef640 10 monclient: dump: 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:27.623 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:27.624 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:27.624 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f716bc55640 10 monclient: shutdown 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f716bc55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f716bc55640 1 -- 172.21.15.55:0/2770000373 >> v1:172.21.15.133:6789/0 conn(0x7f71640b1730 legacy=0x7f71640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f716bc55640 1 -- 172.21.15.55:0/2770000373 shutdown_connections 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.621+0000 7f716bc55640 1 -- 172.21.15.55:0/2770000373 >> 172.21.15.55:0/2770000373 conn(0x7f71640617a0 msgr2=0x7f7164061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.622+0000 7f716bc55640 1 -- 172.21.15.55:0/2770000373 shutdown_connections 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.622+0000 7f716bc55640 1 -- 172.21.15.55:0/2770000373 wait complete. 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.623+0000 7f716bc55640 10 monclient: build_initial_monmap 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.623+0000 7f716bc55640 10 monclient: monmap: 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:27.625 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.623+0000 7f716bc55640 1 librados: starting msgr at 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.623+0000 7f716bc55640 1 librados: starting objecter 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.623+0000 7f716bc55640 1 Processor -- start 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.623+0000 7f716bc55640 1 -- start start 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.623+0000 7f716bc55640 1 librados: setting wanted keys 2024-03-25T18:29:27.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.623+0000 7f716bc55640 1 librados: calling monclient init 2024-03-25T18:29:27.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.623+0000 7f716bc55640 10 monclient: init 2024-03-25T18:29:27.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:27.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:27.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 10 monclient(hunting): picked mon.noname-c con 0x7f71640b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:27.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 10 monclient(hunting): picked mon.noname-a con 0x7f71640b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:27.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 10 monclient(hunting): picked mon.noname-b con 0x7f716414be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:27.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71641440c0 con 0x7f71640b4d60 2024-03-25T18:29:27.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716413f020 con 0x7f716414be90 2024-03-25T18:29:27.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7164136d30 con 0x7f71640b1730 2024-03-25T18:29:27.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:27.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.624+0000 7f716bc55640 10 monclient(hunting): authenticate will time out at 7371.882943s 2024-03-25T18:29:27.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.642+0000 7f716a1f2640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f716414be90 0x7f71641d8260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:53364/0 (socket says 172.21.15.55:53364) 2024-03-25T18:29:27.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.642+0000 7f716a1f2640 1 -- 172.21.15.55:0/4069797143 learned_addr learned my addr 172.21.15.55:0/4069797143 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:27.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.645+0000 7f715a7fc640 10 client.?.objecter ms_handle_connect 0x7f716414be90 2024-03-25T18:29:27.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.645+0000 7f715a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:27.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.646+0000 7f715a7fc640 10 client.?.objecter ms_handle_connect 0x7f71640b1730 2024-03-25T18:29:27.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.646+0000 7f715a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:27.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.651+0000 7f715a7fc640 10 client.?.objecter ms_handle_connect 0x7f71640b4d60 2024-03-25T18:29:27.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.651+0000 7f715a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:27.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.655+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179133231 0 0) 0x7f716413f020 con 0x7f716414be90 2024-03-25T18:29:27.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.655+0000 7f715a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:27.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.655+0000 7f715a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:27.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.655+0000 7f715a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:27.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.655+0000 7f715a7fc640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:29:27.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.655+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7148004eb0 con 0x7f716414be90 2024-03-25T18:29:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.669+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597840813 0 0) 0x7f7164136d30 con 0x7f71640b1730 2024-03-25T18:29:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.669+0000 7f715a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.669+0000 7f715a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.669+0000 7f715a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.669+0000 7f715a7fc640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:29:27.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.669+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f716413f020 con 0x7f71640b1730 2024-03-25T18:29:27.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.672+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163062896 0 0) 0x7f71641440c0 con 0x7f71640b4d60 2024-03-25T18:29:27.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.672+0000 7f715a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:27.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.672+0000 7f715a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:27.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.672+0000 7f715a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:27.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.672+0000 7f715a7fc640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:29:27.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.672+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7164136d30 con 0x7f71640b4d60 2024-03-25T18:29:27.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.683+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2614673579 0 0) 0x7f7148004eb0 con 0x7f716414be90 2024-03-25T18:29:27.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.683+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71641440c0 con 0x7f716414be90 2024-03-25T18:29:27.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.683+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f715c001c90 con 0x7f716414be90 2024-03-25T18:29:27.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.683+0000 7f715a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.683+0000 7f715a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:27.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.683+0000 7f715a7fc640 10 monclient(hunting): dump: 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:27.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.683+0000 7f715a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.688+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3577636547 0 0) 0x7f7164136d30 con 0x7f71640b4d60 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.688+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7148004eb0 con 0x7f71640b4d60 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.688+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7154001b00 con 0x7f71640b4d60 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.688+0000 7f715a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.688+0000 7f715a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.688+0000 7f715a7fc640 10 monclient(hunting): dump: 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.690 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:27.691 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:27.691 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:27.691 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:27.691 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.694+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 681784401 0 0) 0x7f716413f020 con 0x7f71640b1730 2024-03-25T18:29:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.694+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7164136d30 con 0x7f71640b1730 2024-03-25T18:29:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.694+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f71600018b0 con 0x7f71640b1730 2024-03-25T18:29:27.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.694+0000 7f715a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.694+0000 7f715a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.694+0000 7f715a7fc640 10 monclient(hunting): dump: 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:27.697 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:27.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.705+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1907310258 0 0) 0x7f71641440c0 con 0x7f716414be90 2024-03-25T18:29:27.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.705+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 >> v1:172.21.15.55:6790/0 conn(0x7f71640b1730 legacy=0x7f71641439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.705+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 >> v1:172.21.15.55:6789/0 conn(0x7f71640b4d60 legacy=0x7f71641d7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.705+0000 7f715a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:27.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.705+0000 7f715a7fc640 1 monclient: found mon.b 2024-03-25T18:29:27.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.705+0000 7f715a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:27.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.705+0000 7f715a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:27.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.705+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7164136ec0 con 0x7f716414be90 2024-03-25T18:29:27.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f715a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f715a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f715a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f716bc55640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f716bc55640 10 monclient: _renew_subs 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f716bc55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71641e33d0 con 0x7f716414be90 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f716bc55640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f716bc55640 10 monclient: _renew_subs 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f716bc55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71641e3740 con 0x7f716414be90 2024-03-25T18:29:27.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.706+0000 7f716bc55640 1 librados: init done 2024-03-25T18:29:27.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.707+0000 7f716bc55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:27.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.707+0000 7f716bc55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:27.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.707+0000 7f716bc55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:27.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:27.707+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71641d79a0 con 0x7f716414be90 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.001+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f715c003990 con 0x7f716414be90 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.001+0000 7f715a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.001+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f715c003d20 con 0x7f716414be90 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.001+0000 7f715a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.001+0000 7f715a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.001+0000 7f715a7fc640 10 monclient: dump: 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:28.003 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:28.004 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:28.004 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:28.004 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:28.004 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:28.004 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:28.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.001+0000 7f715bfff640 4 set_mon_vals no callback set 2024-03-25T18:29:28.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.001+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f715c003f40 con 0x7f716414be90 2024-03-25T18:29:28.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.002+0000 7f71699f1640 10 monclient: get_auth_request con 0x7f714806e0f0 auth_method 0 2024-03-25T18:29:28.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.002+0000 7f715a7fc640 10 client.4270.objecter ms_handle_connect 0x7f714806e0f0 2024-03-25T18:29:28.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.006+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f715c054a40 con 0x7f716414be90 2024-03-25T18:29:28.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.006+0000 7f715a7fc640 10 client.4270.objecter ms_dispatch 0x7f7164000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:28.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.006+0000 7f715a7fc640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:28.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.006+0000 7f715a7fc640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:28.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.006+0000 7f715a7fc640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T18:29:28.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.007+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f715c050550 con 0x7f716414be90 2024-03-25T18:29:28.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.007+0000 7f715a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:28.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.007+0000 7f715a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:28.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.247+0000 7f716bc55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:28.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.247+0000 7f716bc55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:28.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.247+0000 7f716bc55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:28.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.247+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71641401f0 con 0x7f716414be90 2024-03-25T18:29:28.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.292+0000 7f715a7fc640 1 -- 172.21.15.55:0/4069797143 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f715c052480 con 0x7f716414be90 2024-03-25T18:29:28.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.293+0000 7f715a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:28.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.293+0000 7f715a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:28.295 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:29:28.295 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.294+0000 7f716bc55640 10 librados: watch_flush enter 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.294+0000 7f716bc55640 10 librados: watch_flush exit 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.294+0000 7f716bc55640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.294+0000 7f716bc55640 10 client.4270.objecter successfully canceled tick 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.295+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 >> v1:172.21.15.55:6800/286536064 conn(0x7f714806e0f0 legacy=0x7f71480705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.295+0000 7f716bc55640 10 monclient: shutdown 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.295+0000 7f716bc55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.295+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 >> v1:172.21.15.133:6789/0 conn(0x7f716414be90 legacy=0x7f71641d8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.295+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 shutdown_connections 2024-03-25T18:29:28.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.295+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 >> 172.21.15.55:0/4069797143 conn(0x7f71640617a0 msgr2=0x7f716413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:28.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.295+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 shutdown_connections 2024-03-25T18:29:28.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.295+0000 7f716bc55640 1 -- 172.21.15.55:0/4069797143 wait complete. 2024-03-25T18:29:28.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:28.295+0000 7f716bc55640 1 librados: shutdown 2024-03-25T18:29:28.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:34.295 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:01:44.465 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:34.300 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:01:44.470 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:34.362 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:01:44.532 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:34.368 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:44.506 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:34.373 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:01:44.543 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:34.374 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:01:44.544 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:34.374 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:44.511 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:34.424 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:01:44.561 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:34.432 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:01:44.602 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:35.116 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:35.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.231+0000 7fac4e472640 10 monclient: get_monmap_and_config 2024-03-25T18:29:35.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.231+0000 7fac4e472640 10 monclient: build_initial_monmap 2024-03-25T18:29:35.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.231+0000 7fac4e472640 10 monclient: monmap: 2024-03-25T18:29:35.232 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:35.232 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:35.232 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:29:35.232457+0000 2024-03-25T18:29:35.233 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:29:35.232457+0000 2024-03-25T18:29:35.233 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:35.233 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:35.233 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:35.233 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:29:35.233 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:29:35.233 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:35.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.232+0000 7fac4e472640 1 Processor -- start 2024-03-25T18:29:35.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.232+0000 7fac4e472640 1 -- start start 2024-03-25T18:29:35.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.232+0000 7fac4e472640 10 monclient: init 2024-03-25T18:29:35.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:35.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:35.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 10 monclient(hunting): picked mon.noname-c con 0x7fac480a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:35.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 10 monclient(hunting): picked mon.noname-a con 0x7fac4813f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:35.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 10 monclient(hunting): picked mon.noname-b con 0x7fac481430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:35.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac48142520 con 0x7fac4813f710 2024-03-25T18:29:35.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac480b28a0 con 0x7fac480a98f0 2024-03-25T18:29:35.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac480b2a30 con 0x7fac481430c0 2024-03-25T18:29:35.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:35.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.233+0000 7fac4e472640 10 monclient(hunting): authenticate will time out at 7379.490610s 2024-03-25T18:29:35.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.235+0000 7fac477fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fac4813f710 0x7fac481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:33248/0 (socket says 172.21.15.55:33248) 2024-03-25T18:29:35.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.235+0000 7fac477fe640 1 -- 172.21.15.55:0/115747532 learned_addr learned my addr 172.21.15.55:0/115747532 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:35.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.246+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296853838 0 0) 0x7fac48142520 con 0x7fac4813f710 2024-03-25T18:29:35.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.246+0000 7fac46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:35.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.246+0000 7fac46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:35.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.246+0000 7fac46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:35.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.246+0000 7fac46ffd640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:29:35.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.246+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac30003520 con 0x7fac4813f710 2024-03-25T18:29:35.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.248+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553806232 0 0) 0x7fac480b28a0 con 0x7fac480a98f0 2024-03-25T18:29:35.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.248+0000 7fac46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:35.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.248+0000 7fac46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:35.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.248+0000 7fac46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:35.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.248+0000 7fac46ffd640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:29:35.250 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.248+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac48142520 con 0x7fac480a98f0 2024-03-25T18:29:35.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 538297594 0 0) 0x7fac48142520 con 0x7fac480a98f0 2024-03-25T18:29:35.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 >> v1:172.21.15.55:6790/0 conn(0x7fac481430c0 legacy=0x7fac4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 >> v1:172.21.15.55:6789/0 conn(0x7fac4813f710 legacy=0x7fac481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:35.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 1 monclient: found mon.noname-c 2024-03-25T18:29:35.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:35.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:29:35.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480b44c0 con 0x7fac480a98f0 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac38001670 con 0x7fac480a98f0 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac46ffd640 10 monclient: dump: 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:35.265 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:35.266 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:35.266 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:35.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac4e472640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:29:35.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.263+0000 7fac4e472640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.272+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac380017d0 con 0x7fac480a98f0 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.272+0000 7fac46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.272+0000 7fac46ffd640 1 -- 172.21.15.55:0/115747532 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac38001ba0 con 0x7fac480a98f0 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac46ffd640 10 monclient: dump: 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:35.275 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:35.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac4e472640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:35.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac4e472640 4 set_mon_vals no callback set 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac4e472640 10 monclient: shutdown 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac4e472640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac4e472640 1 -- 172.21.15.55:0/115747532 >> v1:172.21.15.133:6789/0 conn(0x7fac480a98f0 legacy=0x7fac480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac4e472640 1 -- 172.21.15.55:0/115747532 shutdown_connections 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac4e472640 1 -- 172.21.15.55:0/115747532 >> 172.21.15.55:0/115747532 conn(0x7fac480b7ea0 msgr2=0x7fac480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.273+0000 7fac4e472640 1 -- 172.21.15.55:0/115747532 shutdown_connections 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.274+0000 7fac4e472640 1 -- 172.21.15.55:0/115747532 wait complete. 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.274+0000 7fac4e472640 10 monclient: build_initial_monmap 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.274+0000 7fac4e472640 10 monclient: monmap: 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:35.278 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:35.278 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:29:35.278 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:29:35.278 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:35.278 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:29:35.278 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 1 librados: starting msgr at 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 1 librados: starting objecter 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 1 Processor -- start 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 1 -- start start 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 1 librados: setting wanted keys 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 1 librados: calling monclient init 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 10 monclient: init 2024-03-25T18:29:35.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:35.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:35.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 10 monclient(hunting): picked mon.noname-b con 0x7fac480a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:35.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 10 monclient(hunting): picked mon.noname-a con 0x7fac4813f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:35.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.275+0000 7fac4e472640 10 monclient(hunting): picked mon.noname-c con 0x7fac481430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:35.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.276+0000 7fac4e472640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac48139190 con 0x7fac4813f710 2024-03-25T18:29:35.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.276+0000 7fac4e472640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac480aba50 con 0x7fac480a98f0 2024-03-25T18:29:35.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.276+0000 7fac4e472640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac480abd30 con 0x7fac481430c0 2024-03-25T18:29:35.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.276+0000 7fac4e472640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:35.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.276+0000 7fac4e472640 10 monclient(hunting): authenticate will time out at 7379.533446s 2024-03-25T18:29:35.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.279+0000 7fac47fff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fac480a98f0 0x7fac481388c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:51946/0 (socket says 172.21.15.55:51946) 2024-03-25T18:29:35.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.279+0000 7fac47fff640 1 -- 172.21.15.55:0/3027290570 learned_addr learned my addr 172.21.15.55:0/3027290570 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:35.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.280+0000 7fac44ff9640 10 client.?.objecter ms_handle_connect 0x7fac480a98f0 2024-03-25T18:29:35.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.280+0000 7fac44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:35.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.287+0000 7fac44ff9640 10 client.?.objecter ms_handle_connect 0x7fac481430c0 2024-03-25T18:29:35.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.287+0000 7fac44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:35.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.288+0000 7fac44ff9640 10 client.?.objecter ms_handle_connect 0x7fac4813f710 2024-03-25T18:29:35.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.288+0000 7fac44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:35.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.299+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707412497 0 0) 0x7fac48139190 con 0x7fac4813f710 2024-03-25T18:29:35.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.299+0000 7fac44ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:35.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.299+0000 7fac44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:35.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.299+0000 7fac44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:35.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.299+0000 7fac44ff9640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:29:35.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.299+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac34004eb0 con 0x7fac4813f710 2024-03-25T18:29:35.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.378+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149636348 0 0) 0x7fac480abd30 con 0x7fac481430c0 2024-03-25T18:29:35.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.378+0000 7fac44ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:35.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.378+0000 7fac44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:35.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.378+0000 7fac44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:35.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.378+0000 7fac44ff9640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:29:35.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.378+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac48139190 con 0x7fac481430c0 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.380+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 664765679 0 0) 0x7fac34004eb0 con 0x7fac4813f710 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.380+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac480abd30 con 0x7fac4813f710 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.380+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac3c001950 con 0x7fac4813f710 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.380+0000 7fac44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.380+0000 7fac44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.380+0000 7fac44ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.381 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.382 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.382 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:35.382 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:35.382 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:35.382 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:35.382 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:35.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.380+0000 7fac44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:35.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.392+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438293880 0 0) 0x7fac48139190 con 0x7fac481430c0 2024-03-25T18:29:35.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.393+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac34004eb0 con 0x7fac481430c0 2024-03-25T18:29:35.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.393+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac40001b10 con 0x7fac481430c0 2024-03-25T18:29:35.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.393+0000 7fac44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.393+0000 7fac44ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:35.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.393+0000 7fac44ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:35.395 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:35.395 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:35.395 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.442 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.442 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.442 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:35.443 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:35.443 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:35.443 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:35.443 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.400+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2109435431 0 0) 0x7fac34004eb0 con 0x7fac481430c0 2024-03-25T18:29:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.400+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 >> v1:172.21.15.133:6789/0 conn(0x7fac480a98f0 legacy=0x7fac481388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.400+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 >> v1:172.21.15.55:6789/0 conn(0x7fac4813f710 legacy=0x7fac480ac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.400+0000 7fac44ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.400+0000 7fac44ff9640 1 monclient: found mon.c 2024-03-25T18:29:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:35.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac44ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac48137d20 con 0x7fac481430c0 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac44ff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac44ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 5 monclient: authenticate success, global_id 4292 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 10 monclient: _renew_subs 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac481364f0 con 0x7fac481430c0 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 10 monclient: _renew_subs 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac481e39b0 con 0x7fac481430c0 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 1 librados: init done 2024-03-25T18:29:35.445 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.401+0000 7fac4e472640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:35.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.402+0000 7fac4e472640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:35.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.402+0000 7fac4e472640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:35.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.402+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac480b3ef0 con 0x7fac481430c0 2024-03-25T18:29:35.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.714+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac40002000 con 0x7fac481430c0 2024-03-25T18:29:35.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.714+0000 7fac44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.714+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac40003b10 con 0x7fac481430c0 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.714+0000 7fac44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.714+0000 7fac44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.714+0000 7fac44ff9640 10 monclient: dump: 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:35.716 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:35.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.714+0000 7fac467fc640 4 set_mon_vals no callback set 2024-03-25T18:29:35.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.715+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fac40003d30 con 0x7fac481430c0 2024-03-25T18:29:35.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.716+0000 7fac47fff640 10 monclient: get_auth_request con 0x7fac3406de90 auth_method 0 2024-03-25T18:29:35.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.716+0000 7fac44ff9640 10 client.4292.objecter ms_handle_connect 0x7fac3406de90 2024-03-25T18:29:35.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.724+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fac40053800 con 0x7fac481430c0 2024-03-25T18:29:35.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.724+0000 7fac44ff9640 10 client.4292.objecter ms_dispatch 0x7fac48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:35.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.724+0000 7fac44ff9640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:35.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.724+0000 7fac44ff9640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:35.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.724+0000 7fac44ff9640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-25T18:29:35.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.724+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac4004f310 con 0x7fac481430c0 2024-03-25T18:29:35.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.725+0000 7fac44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:35.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.725+0000 7fac44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:35.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.976+0000 7fac4e472640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:35.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.976+0000 7fac4e472640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:35.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.976+0000 7fac4e472640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:29:35.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:35.976+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac48137eb0 con 0x7fac481430c0 2024-03-25T18:29:36.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.002+0000 7fac44ff9640 1 -- 172.21.15.55:0/3027290570 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fac40051240 con 0x7fac481430c0 2024-03-25T18:29:36.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.003+0000 7fac44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:36.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.003+0000 7fac44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:36.004 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:29:36.004 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:36.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 10 librados: watch_flush enter 2024-03-25T18:29:36.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 10 librados: watch_flush exit 2024-03-25T18:29:36.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-25T18:29:36.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 10 client.4292.objecter successfully canceled tick 2024-03-25T18:29:36.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 >> v1:172.21.15.55:6800/286536064 conn(0x7fac3406de90 legacy=0x7fac34070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:36.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 10 monclient: shutdown 2024-03-25T18:29:36.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:36.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 >> v1:172.21.15.55:6790/0 conn(0x7fac481430c0 legacy=0x7fac481e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:36.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 shutdown_connections 2024-03-25T18:29:36.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 >> 172.21.15.55:0/3027290570 conn(0x7fac480b7ea0 msgr2=0x7fac48074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:36.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 shutdown_connections 2024-03-25T18:29:36.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 1 -- 172.21.15.55:0/3027290570 wait complete. 2024-03-25T18:29:36.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:36.005+0000 7fac4e472640 1 librados: shutdown 2024-03-25T18:29:36.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:42.825 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:42.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.938+0000 7f587f37b640 10 monclient: get_monmap_and_config 2024-03-25T18:29:42.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.938+0000 7f587f37b640 10 monclient: build_initial_monmap 2024-03-25T18:29:42.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.938+0000 7f587f37b640 10 monclient: monmap: 2024-03-25T18:29:42.939 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:42.939 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:42.940 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:29:42.939311+0000 2024-03-25T18:29:42.940 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:29:42.939311+0000 2024-03-25T18:29:42.940 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:42.940 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:42.940 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:42.940 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:29:42.940 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:29:42.940 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 1 Processor -- start 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 1 -- start start 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 10 monclient: init 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 10 monclient(hunting): picked mon.noname-a con 0x7f58780b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 10 monclient(hunting): picked mon.noname-c con 0x7f58780b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 10 monclient(hunting): picked mon.noname-b con 0x7f587814be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58780af940 con 0x7f58780b4d60 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58780b0a70 con 0x7f58780b1730 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58780b0d50 con 0x7f587814be90 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.939+0000 7f587f37b640 10 monclient(hunting): authenticate will time out at 7387.197345s 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.946+0000 7f587d918640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f587814be90 0x7f587814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:54212/0 (socket says 172.21.15.55:54212) 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.946+0000 7f587d918640 1 -- 172.21.15.55:0/1185946445 learned_addr learned my addr 172.21.15.55:0/1185946445 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:42.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.961+0000 7f5877fff640 1 -- 172.21.15.55:0/1185946445 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457680254 0 0) 0x7f58780b0a70 con 0x7f58780b1730 2024-03-25T18:29:42.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.961+0000 7f5877fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.961+0000 7f5877fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.961+0000 7f5877fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.961+0000 7f5877fff640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:29:42.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.961+0000 7f5877fff640 1 -- 172.21.15.55:0/1185946445 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5860003520 con 0x7f58780b1730 2024-03-25T18:29:42.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 1 -- 172.21.15.55:0/1185946445 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3144049279 0 0) 0x7f5860003520 con 0x7f58780b1730 2024-03-25T18:29:42.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 1 -- 172.21.15.55:0/1185946445 >> v1:172.21.15.55:6790/0 conn(0x7f587814be90 legacy=0x7f587814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 1 -- 172.21.15.55:0/1185946445 >> v1:172.21.15.55:6789/0 conn(0x7f58780b4d60 legacy=0x7f587813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:29:42.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:42.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 1 monclient: found mon.noname-c 2024-03-25T18:29:42.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 1 -- 172.21.15.55:0/1185946445 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58780b1030 con 0x7f58780b1730 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 1 -- 172.21.15.55:0/1185946445 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f586c001690 con 0x7f58780b1730 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f587f37b640 5 monclient: authenticate success, global_id 4279 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f587f37b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f5877fff640 10 monclient: dump: 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:42.979 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:42.980 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.980 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:42.980 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:42.980 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:42.980 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:42.980 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:42.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.977+0000 7f587f37b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.994+0000 7f5877fff640 1 -- 172.21.15.55:0/1185946445 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f586c001860 con 0x7f58780b1730 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.994+0000 7f5877fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.994+0000 7f5877fff640 1 -- 172.21.15.55:0/1185946445 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f586c001cb0 con 0x7f58780b1730 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.994+0000 7f587f37b640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.994+0000 7f587f37b640 4 set_mon_vals no callback set 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.994+0000 7f5877fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.994+0000 7f5877fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.994+0000 7f5877fff640 10 monclient: dump: 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:42.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.995+0000 7f587f37b640 10 monclient: shutdown 2024-03-25T18:29:42.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.995+0000 7f587f37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:42.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.995+0000 7f587f37b640 1 -- 172.21.15.55:0/1185946445 >> v1:172.21.15.133:6789/0 conn(0x7f58780b1730 legacy=0x7f58780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.995+0000 7f587f37b640 1 -- 172.21.15.55:0/1185946445 shutdown_connections 2024-03-25T18:29:42.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.995+0000 7f587f37b640 1 -- 172.21.15.55:0/1185946445 >> 172.21.15.55:0/1185946445 conn(0x7f58780617a0 msgr2=0x7f5878061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.996+0000 7f587f37b640 1 -- 172.21.15.55:0/1185946445 shutdown_connections 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.996+0000 7f587f37b640 1 -- 172.21.15.55:0/1185946445 wait complete. 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.996+0000 7f587f37b640 10 monclient: build_initial_monmap 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.996+0000 7f587f37b640 10 monclient: monmap: 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:43.000 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.997+0000 7f587f37b640 1 librados: starting msgr at 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.997+0000 7f587f37b640 1 librados: starting objecter 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.997+0000 7f587f37b640 1 Processor -- start 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.997+0000 7f587f37b640 1 -- start start 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.997+0000 7f587f37b640 1 librados: setting wanted keys 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.997+0000 7f587f37b640 1 librados: calling monclient init 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.997+0000 7f587f37b640 10 monclient: init 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 10 monclient(hunting): picked mon.noname-a con 0x7f58780b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:43.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 10 monclient(hunting): picked mon.noname-b con 0x7f58780b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:43.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 10 monclient(hunting): picked mon.noname-c con 0x7f587814be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:43.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58781d7620 con 0x7f58780b1730 2024-03-25T18:29:43.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58781d7870 con 0x7f58780b4d60 2024-03-25T18:29:43.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58781d8480 con 0x7f587814be90 2024-03-25T18:29:43.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:43.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:42.998+0000 7f587f37b640 10 monclient(hunting): authenticate will time out at 7387.255753s 2024-03-25T18:29:43.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.001+0000 7f587d918640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f587814be90 0x7f5878142b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:54234/0 (socket says 172.21.15.55:54234) 2024-03-25T18:29:43.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.001+0000 7f587d918640 1 -- 172.21.15.55:0/2890941254 learned_addr learned my addr 172.21.15.55:0/2890941254 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:43.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.002+0000 7f5875ffb640 10 client.?.objecter ms_handle_connect 0x7f587814be90 2024-03-25T18:29:43.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.002+0000 7f5875ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:43.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.007+0000 7f5875ffb640 10 client.?.objecter ms_handle_connect 0x7f58780b4d60 2024-03-25T18:29:43.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.007+0000 7f5875ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:43.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.061+0000 7f5875ffb640 10 client.?.objecter ms_handle_connect 0x7f58780b1730 2024-03-25T18:29:43.062 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.061+0000 7f5875ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:43.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.074+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659720622 0 0) 0x7f58781d7620 con 0x7f58780b1730 2024-03-25T18:29:43.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.074+0000 7f5875ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:43.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.074+0000 7f5875ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:43.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.074+0000 7f5875ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:43.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.074+0000 7f5875ffb640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:29:43.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.074+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5864004eb0 con 0x7f58780b1730 2024-03-25T18:29:43.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.092+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2300322893 0 0) 0x7f5864004eb0 con 0x7f58780b1730 2024-03-25T18:29:43.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.092+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58781d7620 con 0x7f58780b1730 2024-03-25T18:29:43.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.092+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f5868001bf0 con 0x7f58780b1730 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.092+0000 7f5875ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.092+0000 7f5875ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.092+0000 7f5875ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:43.095 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:43.096 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:43.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.092+0000 7f5875ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:43.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3152903943 0 0) 0x7f58781d7620 con 0x7f58780b1730 2024-03-25T18:29:43.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 >> v1:172.21.15.55:6790/0 conn(0x7f587814be90 legacy=0x7f5878142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:43.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 >> v1:172.21.15.133:6789/0 conn(0x7f58780b4d60 legacy=0x7f58781d7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:43.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:43.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 1 monclient: found mon.a 2024-03-25T18:29:43.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:43.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:43.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587813ba00 con 0x7f58780b1730 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f5875ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f587f37b640 5 monclient: authenticate success, global_id 4299 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f587f37b640 10 monclient: _renew_subs 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f587f37b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587813bbf0 con 0x7f58780b1730 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f587f37b640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f587f37b640 10 monclient: _renew_subs 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f587f37b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5878142430 con 0x7f58780b1730 2024-03-25T18:29:43.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.114+0000 7f587f37b640 1 librados: init done 2024-03-25T18:29:43.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.115+0000 7f587f37b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:43.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.115+0000 7f587f37b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:43.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.115+0000 7f587f37b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:43.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.115+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58780627b0 con 0x7f58780b1730 2024-03-25T18:29:43.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.431+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5868003970 con 0x7f58780b1730 2024-03-25T18:29:43.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.431+0000 7f5875ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:43.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.431+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f5868003c80 con 0x7f58780b1730 2024-03-25T18:29:43.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.431+0000 7f5875ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.431+0000 7f5875ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.431+0000 7f5875ffb640 10 monclient: dump: 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:43.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.431+0000 7f58777fe640 4 set_mon_vals no callback set 2024-03-25T18:29:43.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.432+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f5868019c50 con 0x7f58780b1730 2024-03-25T18:29:43.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.433+0000 7f587c916640 10 monclient: get_auth_request con 0x7f586406dd00 auth_method 0 2024-03-25T18:29:43.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.433+0000 7f5875ffb640 10 client.4299.objecter ms_handle_connect 0x7f586406dd00 2024-03-25T18:29:43.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.440+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f5868053920 con 0x7f58780b1730 2024-03-25T18:29:43.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.440+0000 7f5875ffb640 10 client.4299.objecter ms_dispatch 0x7f5878000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:43.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.440+0000 7f5875ffb640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:43.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.440+0000 7f5875ffb640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:43.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.440+0000 7f5875ffb640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-25T18:29:43.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.440+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f586804f430 con 0x7f58780b1730 2024-03-25T18:29:43.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.440+0000 7f5875ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:43.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.440+0000 7f5875ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:43.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.697+0000 7f587f37b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:43.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.697+0000 7f587f37b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:43.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.697+0000 7f587f37b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:43.698 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.697+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5878069f40 con 0x7f58780b1730 2024-03-25T18:29:43.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.734+0000 7f5875ffb640 1 -- 172.21.15.55:0/2890941254 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f5868051360 con 0x7f58780b1730 2024-03-25T18:29:43.736 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.734+0000 7f5875ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:43.736 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.734+0000 7f5875ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:43.736 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:29:43.736 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:43.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 10 librados: watch_flush enter 2024-03-25T18:29:43.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 10 librados: watch_flush exit 2024-03-25T18:29:43.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-25T18:29:43.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 10 client.4299.objecter successfully canceled tick 2024-03-25T18:29:43.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 >> v1:172.21.15.55:6800/286536064 conn(0x7f586406dd00 legacy=0x7f58640701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:43.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 10 monclient: shutdown 2024-03-25T18:29:43.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:43.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 >> v1:172.21.15.55:6789/0 conn(0x7f58780b1730 legacy=0x7f58781405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:43.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 shutdown_connections 2024-03-25T18:29:43.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 >> 172.21.15.55:0/2890941254 conn(0x7f58780617a0 msgr2=0x7f58780b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:43.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 shutdown_connections 2024-03-25T18:29:43.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 1 -- 172.21.15.55:0/2890941254 wait complete. 2024-03-25T18:29:43.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:43.737+0000 7f587f37b640 1 librados: shutdown 2024-03-25T18:29:43.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:50.557 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:50.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.674+0000 7ff3c0e3e640 10 monclient: get_monmap_and_config 2024-03-25T18:29:50.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.674+0000 7ff3c0e3e640 10 monclient: build_initial_monmap 2024-03-25T18:29:50.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.674+0000 7ff3c0e3e640 10 monclient: monmap: 2024-03-25T18:29:50.675 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:50.675 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:50.675 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:29:50.675146+0000 2024-03-25T18:29:50.675 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:29:50.675146+0000 2024-03-25T18:29:50.676 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:50.676 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:50.676 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:50.676 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:29:50.676 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:29:50.676 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:50.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.675+0000 7ff3c0e3e640 1 Processor -- start 2024-03-25T18:29:50.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.675+0000 7ff3c0e3e640 1 -- start start 2024-03-25T18:29:50.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.675+0000 7ff3c0e3e640 10 monclient: init 2024-03-25T18:29:50.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:50.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:50.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3bc0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:50.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3bc13f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:50.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3bc1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:50.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc142520 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc0b28a0 con 0x7ff3bc1430c0 2024-03-25T18:29:50.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc0b2a30 con 0x7ff3bc13f710 2024-03-25T18:29:50.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:50.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.676+0000 7ff3c0e3e640 10 monclient(hunting): authenticate will time out at 7394.933739s 2024-03-25T18:29:50.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.684+0000 7ff3bad9d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7ff3bc1430c0 0x7ff3bc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:39920/0 (socket says 172.21.15.55:39920) 2024-03-25T18:29:50.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.684+0000 7ff3bad9d640 1 -- 172.21.15.55:0/597224362 learned_addr learned my addr 172.21.15.55:0/597224362 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:50.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.698+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424521132 0 0) 0x7ff3bc142520 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.698+0000 7ff3b959a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:50.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.698+0000 7ff3b959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:50.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.698+0000 7ff3b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:50.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.698+0000 7ff3b959a640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:29:50.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.698+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a4003520 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.700+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926126859 0 0) 0x7ff3bc0b28a0 con 0x7ff3bc1430c0 2024-03-25T18:29:50.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.700+0000 7ff3b959a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:50.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.700+0000 7ff3b959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:50.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.700+0000 7ff3b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:50.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.700+0000 7ff3b959a640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:29:50.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.700+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3bc142520 con 0x7ff3bc1430c0 2024-03-25T18:29:50.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2205552045 0 0) 0x7ff3a4003520 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 >> v1:172.21.15.55:6790/0 conn(0x7ff3bc13f710 legacy=0x7ff3bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 >> v1:172.21.15.133:6789/0 conn(0x7ff3bc1430c0 legacy=0x7ff3bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 10 monclient: _finish_hunting 0 2024-03-25T18:29:50.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 1 monclient: found mon.noname-a 2024-03-25T18:29:50.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc0b44c0 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 10 monclient: _finish_auth 0 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 10 monclient: _check_auth_tickets 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3ac001690 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3b959a640 10 monclient: dump: 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.712 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:50.713 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:50.713 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:50.713 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:50.713 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:50.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3c0e3e640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:29:50.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.710+0000 7ff3c0e3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3ac0017f0 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3b959a640 1 -- 172.21.15.55:0/597224362 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3ac001c00 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3b959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3b959a640 10 monclient: dump: 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.718 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:50.719 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:50.719 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:50.719 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:50.719 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:50.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3c0e3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:50.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3c0e3e640 4 set_mon_vals no callback set 2024-03-25T18:29:50.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3c0e3e640 10 monclient: shutdown 2024-03-25T18:29:50.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3c0e3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:50.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.717+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/597224362 >> v1:172.21.15.55:6789/0 conn(0x7ff3bc0a98f0 legacy=0x7ff3bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.718+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/597224362 shutdown_connections 2024-03-25T18:29:50.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.718+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/597224362 >> 172.21.15.55:0/597224362 conn(0x7ff3bc0b7ea0 msgr2=0x7ff3bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.718+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/597224362 shutdown_connections 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.718+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/597224362 wait complete. 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.719+0000 7ff3c0e3e640 10 monclient: build_initial_monmap 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.719+0000 7ff3c0e3e640 10 monclient: monmap: 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:29:50.721 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:50.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.719+0000 7ff3c0e3e640 1 librados: starting msgr at 2024-03-25T18:29:50.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.719+0000 7ff3c0e3e640 1 librados: starting objecter 2024-03-25T18:29:50.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.719+0000 7ff3c0e3e640 1 Processor -- start 2024-03-25T18:29:50.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.719+0000 7ff3c0e3e640 1 -- start start 2024-03-25T18:29:50.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.719+0000 7ff3c0e3e640 1 librados: setting wanted keys 2024-03-25T18:29:50.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.719+0000 7ff3c0e3e640 1 librados: calling monclient init 2024-03-25T18:29:50.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.719+0000 7ff3c0e3e640 10 monclient: init 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3bc0a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3bc13f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3bc1430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc132a30 con 0x7ff3bc1430c0 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc138c80 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc138f60 con 0x7ff3bc13f710 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:50.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.720+0000 7ff3c0e3e640 10 monclient(hunting): authenticate will time out at 7394.978143s 2024-03-25T18:29:50.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.728+0000 7ff3bad9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ff3bc1430c0 0x7ff3bc132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:37478/0 (socket says 172.21.15.55:37478) 2024-03-25T18:29:50.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.728+0000 7ff3bad9d640 1 -- 172.21.15.55:0/1393424779 learned_addr learned my addr 172.21.15.55:0/1393424779 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:50.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.729+0000 7ff3aaffd640 10 client.?.objecter ms_handle_connect 0x7ff3bc1430c0 2024-03-25T18:29:50.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.729+0000 7ff3aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:50.737 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.737+0000 7ff3aaffd640 10 client.?.objecter ms_handle_connect 0x7ff3bc0a98f0 2024-03-25T18:29:50.737 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.737+0000 7ff3aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:50.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.737+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695896505 0 0) 0x7ff3bc132a30 con 0x7ff3bc1430c0 2024-03-25T18:29:50.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.737+0000 7ff3aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:50.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.737+0000 7ff3aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:50.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.737+0000 7ff3aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:50.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.737+0000 7ff3aaffd640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:29:50.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.737+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c004eb0 con 0x7ff3bc1430c0 2024-03-25T18:29:50.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.742+0000 7ff3aaffd640 10 client.?.objecter ms_handle_connect 0x7ff3bc13f710 2024-03-25T18:29:50.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.742+0000 7ff3aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:50.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.751+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558234465 0 0) 0x7ff3bc138c80 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.751+0000 7ff3aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:50.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.751+0000 7ff3aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:50.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.751+0000 7ff3aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:50.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.751+0000 7ff3aaffd640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:29:50.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.751+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3bc132a30 con 0x7ff3bc0a98f0 2024-03-25T18:29:50.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.753+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723000576 0 0) 0x7ff3bc138f60 con 0x7ff3bc13f710 2024-03-25T18:29:50.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.753+0000 7ff3aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:50.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.753+0000 7ff3aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:50.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.753+0000 7ff3aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:50.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.753+0000 7ff3aaffd640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:29:50.755 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.753+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3bc138c80 con 0x7ff3bc13f710 2024-03-25T18:29:50.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.756+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3998809214 0 0) 0x7ff39c004eb0 con 0x7ff3bc1430c0 2024-03-25T18:29:50.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.756+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3bc138f60 con 0x7ff3bc1430c0 2024-03-25T18:29:50.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.756+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3b4001c10 con 0x7ff3bc1430c0 2024-03-25T18:29:50.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.756+0000 7ff3aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.756+0000 7ff3aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.756+0000 7ff3aaffd640 10 monclient(hunting): dump: 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:50.758 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:50.759 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:50.759 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:50.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.756+0000 7ff3aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:50.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.771+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2095502133 0 0) 0x7ff3bc138c80 con 0x7ff3bc13f710 2024-03-25T18:29:50.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.771+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff39c004eb0 con 0x7ff3bc13f710 2024-03-25T18:29:50.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.771+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3b0001870 con 0x7ff3bc13f710 2024-03-25T18:29:50.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.771+0000 7ff3aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.771+0000 7ff3aaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:50.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.771+0000 7ff3aaffd640 10 monclient(hunting): dump: 2024-03-25T18:29:50.775 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:50.775 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:50.775 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:50.775 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:50.775 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.776 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:50.776 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:50.776 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:50.776 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:50.776 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:50.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 307545532 0 0) 0x7ff3bc138f60 con 0x7ff3bc1430c0 2024-03-25T18:29:50.778 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 >> v1:172.21.15.55:6790/0 conn(0x7ff3bc13f710 legacy=0x7ff3bc131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 >> v1:172.21.15.133:6789/0 conn(0x7ff3bc0a98f0 legacy=0x7ff3bc137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:50.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 1 monclient: found mon.a 2024-03-25T18:29:50.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:50.779 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:50.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc139240 con 0x7ff3bc1430c0 2024-03-25T18:29:50.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:50.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:50.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.772+0000 7ff3aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:50.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.773+0000 7ff3c0e3e640 5 monclient: authenticate success, global_id 4305 2024-03-25T18:29:50.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.773+0000 7ff3c0e3e640 10 monclient: _renew_subs 2024-03-25T18:29:50.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.773+0000 7ff3c0e3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:50.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.773+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3bc136e50 con 0x7ff3bc1430c0 2024-03-25T18:29:50.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.773+0000 7ff3c0e3e640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:50.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.773+0000 7ff3c0e3e640 10 monclient: _renew_subs 2024-03-25T18:29:50.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.773+0000 7ff3c0e3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:50.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.773+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3bc1389f0 con 0x7ff3bc1430c0 2024-03-25T18:29:50.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.773+0000 7ff3c0e3e640 1 librados: init done 2024-03-25T18:29:50.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.774+0000 7ff3c0e3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:50.782 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.774+0000 7ff3c0e3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:50.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.774+0000 7ff3c0e3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:50.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.774+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3bc0627b0 con 0x7ff3bc1430c0 2024-03-25T18:29:50.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.782+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3b4004b60 con 0x7ff3bc1430c0 2024-03-25T18:29:50.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.782+0000 7ff3aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:50.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:50.782+0000 7ff3b8d99640 4 set_mon_vals no callback set 2024-03-25T18:29:51.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.114+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3b4004e70 con 0x7ff3bc1430c0 2024-03-25T18:29:51.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.114+0000 7ff3aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:51.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.114+0000 7ff3aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:51.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.114+0000 7ff3aaffd640 10 monclient: dump: 2024-03-25T18:29:51.115 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:51.115 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:51.115 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:51.115 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:51.116 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:51.116 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:51.116 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:51.116 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:51.116 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:51.116 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:51.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.115+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7ff3b401ae40 con 0x7ff3bc1430c0 2024-03-25T18:29:51.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.115+0000 7ff3ba59c640 10 monclient: get_auth_request con 0x7ff39c06df70 auth_method 0 2024-03-25T18:29:51.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.115+0000 7ff3aaffd640 10 client.4305.objecter ms_handle_connect 0x7ff39c06df70 2024-03-25T18:29:51.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.122+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7ff3b4054b40 con 0x7ff3bc1430c0 2024-03-25T18:29:51.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.122+0000 7ff3aaffd640 10 client.4305.objecter ms_dispatch 0x7ff3bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:51.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.122+0000 7ff3aaffd640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:51.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.122+0000 7ff3aaffd640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:51.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.122+0000 7ff3aaffd640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-25T18:29:51.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.122+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3b4050480 con 0x7ff3bc1430c0 2024-03-25T18:29:51.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.122+0000 7ff3aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:51.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.122+0000 7ff3aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:51.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.375+0000 7ff3c0e3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:51.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.375+0000 7ff3c0e3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:51.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.375+0000 7ff3c0e3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:29:51.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.375+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3bc13dd60 con 0x7ff3bc1430c0 2024-03-25T18:29:51.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.397+0000 7ff3aaffd640 1 -- 172.21.15.55:0/1393424779 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7ff3b4052580 con 0x7ff3bc1430c0 2024-03-25T18:29:51.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.397+0000 7ff3aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:51.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.397+0000 7ff3aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:51.399 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:29:51.399 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:51.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 10 librados: watch_flush enter 2024-03-25T18:29:51.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 10 librados: watch_flush exit 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 10 client.4305.objecter successfully canceled tick 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 >> v1:172.21.15.55:6800/286536064 conn(0x7ff39c06df70 legacy=0x7ff39c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 10 monclient: shutdown 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 >> v1:172.21.15.55:6789/0 conn(0x7ff3bc1430c0 legacy=0x7ff3bc132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 shutdown_connections 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 >> 172.21.15.55:0/1393424779 conn(0x7ff3bc0b7ea0 msgr2=0x7ff3bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 shutdown_connections 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 1 -- 172.21.15.55:0/1393424779 wait complete. 2024-03-25T18:29:51.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:51.399+0000 7ff3c0e3e640 1 librados: shutdown 2024-03-25T18:29:51.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:54.303 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:02:04.472 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:54.370 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:02:04.538 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:54.399 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:04.536 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:54.400 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:02:04.570 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:54.401 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:02:04.571 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:54.410 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:04.547 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:54.414 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:02:04.584 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:54.442 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:02:04.612 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:54.444 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:04.581 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:58.219 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:58.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.334+0000 7fdd1757b640 10 monclient: get_monmap_and_config 2024-03-25T18:29:58.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.334+0000 7fdd1757b640 10 monclient: build_initial_monmap 2024-03-25T18:29:58.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.334+0000 7fdd1757b640 10 monclient: monmap: 2024-03-25T18:29:58.336 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:58.336 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:58.336 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:29:58.335796+0000 2024-03-25T18:29:58.336 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:29:58.335796+0000 2024-03-25T18:29:58.336 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:58.337 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:58.337 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:58.337 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:29:58.337 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:29:58.337 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:58.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.335+0000 7fdd1757b640 1 Processor -- start 2024-03-25T18:29:58.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.335+0000 7fdd1757b640 1 -- start start 2024-03-25T18:29:58.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.335+0000 7fdd1757b640 10 monclient: init 2024-03-25T18:29:58.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:58.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:58.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd100a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:58.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:58.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 10 monclient(hunting): picked mon.noname-a con 0x7fdd101430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:58.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd10142520 con 0x7fdd101430c0 2024-03-25T18:29:58.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd100b28a0 con 0x7fdd100a98f0 2024-03-25T18:29:58.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd100b2a30 con 0x7fdd1013f710 2024-03-25T18:29:58.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:58.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.336+0000 7fdd1757b640 10 monclient(hunting): authenticate will time out at 7402.593821s 2024-03-25T18:29:58.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.347+0000 7fdd15b18640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fdd101430c0 0x7fdd1014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:42154/0 (socket says 172.21.15.55:42154) 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.347+0000 7fdd15b18640 1 -- 172.21.15.55:0/3503266280 learned_addr learned my addr 172.21.15.55:0/3503266280 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:58.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.358+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025297062 0 0) 0x7fdd10142520 con 0x7fdd101430c0 2024-03-25T18:29:58.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.358+0000 7fdd07fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.358+0000 7fdd07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.358+0000 7fdd07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.358+0000 7fdd07fff640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:29:58.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.358+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcfc003520 con 0x7fdd101430c0 2024-03-25T18:29:58.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.365+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318324107 0 0) 0x7fdd100b2a30 con 0x7fdd1013f710 2024-03-25T18:29:58.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.365+0000 7fdd07fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.365+0000 7fdd07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.365+0000 7fdd07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.365+0000 7fdd07fff640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:29:58.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.365+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd10142520 con 0x7fdd1013f710 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.369+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131297591 0 0) 0x7fdd100b28a0 con 0x7fdd100a98f0 2024-03-25T18:29:58.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.369+0000 7fdd07fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.369+0000 7fdd07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.369+0000 7fdd07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.369+0000 7fdd07fff640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:29:58.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.369+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd100b2a30 con 0x7fdd100a98f0 2024-03-25T18:29:58.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776869825 0 0) 0x7fdd10142520 con 0x7fdd1013f710 2024-03-25T18:29:58.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 >> v1:172.21.15.133:6789/0 conn(0x7fdd100a98f0 legacy=0x7fdd100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 >> v1:172.21.15.55:6789/0 conn(0x7fdd101430c0 legacy=0x7fdd1014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:58.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 1 monclient: found mon.noname-b 2024-03-25T18:29:58.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:58.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:29:58.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd100b44c0 con 0x7fdd1013f710 2024-03-25T18:29:58.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:58.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:58.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:58.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd1757b640 5 monclient: authenticate success, global_id 4304 2024-03-25T18:29:58.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.371+0000 7fdd1757b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.372+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdd00001630 con 0x7fdd1013f710 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.372+0000 7fdd07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.372+0000 7fdd07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.372+0000 7fdd07fff640 10 monclient: dump: 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.374 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:58.375 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:58.375 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:58.375 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:58.375 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:58.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.372+0000 7fdd1757b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:58.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.380+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd00001880 con 0x7fdd1013f710 2024-03-25T18:29:58.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.380+0000 7fdd07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:58.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.380+0000 7fdd1757b640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:58.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.380+0000 7fdd1757b640 4 set_mon_vals no callback set 2024-03-25T18:29:58.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.380+0000 7fdd07fff640 1 -- 172.21.15.55:0/3503266280 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdd00001c10 con 0x7fdd1013f710 2024-03-25T18:29:58.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.380+0000 7fdd07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.380+0000 7fdd07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.380+0000 7fdd07fff640 10 monclient: dump: 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:58.382 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.381+0000 7fdd1757b640 10 monclient: shutdown 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.381+0000 7fdd1757b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.381+0000 7fdd1757b640 1 -- 172.21.15.55:0/3503266280 >> v1:172.21.15.55:6790/0 conn(0x7fdd1013f710 legacy=0x7fdd101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.381+0000 7fdd1757b640 1 -- 172.21.15.55:0/3503266280 shutdown_connections 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.381+0000 7fdd1757b640 1 -- 172.21.15.55:0/3503266280 >> 172.21.15.55:0/3503266280 conn(0x7fdd100b7ea0 msgr2=0x7fdd100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.381+0000 7fdd1757b640 1 -- 172.21.15.55:0/3503266280 shutdown_connections 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.381+0000 7fdd1757b640 1 -- 172.21.15.55:0/3503266280 wait complete. 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.382+0000 7fdd1757b640 10 monclient: build_initial_monmap 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.382+0000 7fdd1757b640 10 monclient: monmap: 2024-03-25T18:29:58.385 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.382+0000 7fdd1757b640 1 librados: starting msgr at 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.382+0000 7fdd1757b640 1 librados: starting objecter 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.382+0000 7fdd1757b640 1 Processor -- start 2024-03-25T18:29:58.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.382+0000 7fdd1757b640 1 -- start start 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.382+0000 7fdd1757b640 1 librados: setting wanted keys 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.382+0000 7fdd1757b640 1 librados: calling monclient init 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.382+0000 7fdd1757b640 10 monclient: init 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd100a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 10 monclient(hunting): picked mon.noname-b con 0x7fdd101430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd101379b0 con 0x7fdd1013f710 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd101db7f0 con 0x7fdd101430c0 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd10137c00 con 0x7fdd100a98f0 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.383+0000 7fdd1757b640 10 monclient(hunting): authenticate will time out at 7402.641300s 2024-03-25T18:29:58.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.395+0000 7fdd15b18640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fdd101430c0 0x7fdd101e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:44626/0 (socket says 172.21.15.55:44626) 2024-03-25T18:29:58.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.396+0000 7fdd15b18640 1 -- 172.21.15.55:0/2554545434 learned_addr learned my addr 172.21.15.55:0/2554545434 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:29:58.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.396+0000 7fdd05ffb640 10 client.?.objecter ms_handle_connect 0x7fdd101430c0 2024-03-25T18:29:58.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.396+0000 7fdd05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:58.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.403+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511795980 0 0) 0x7fdd101db7f0 con 0x7fdd101430c0 2024-03-25T18:29:58.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.403+0000 7fdd05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.403+0000 7fdd05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.403+0000 7fdd05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.403+0000 7fdd05ffb640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:29:58.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.403+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf4004eb0 con 0x7fdd101430c0 2024-03-25T18:29:58.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.404+0000 7fdd05ffb640 10 client.?.objecter ms_handle_connect 0x7fdd1013f710 2024-03-25T18:29:58.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.404+0000 7fdd05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:58.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.405+0000 7fdd05ffb640 10 client.?.objecter ms_handle_connect 0x7fdd100a98f0 2024-03-25T18:29:58.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.405+0000 7fdd05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:58.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.417+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843170317 0 0) 0x7fdd101379b0 con 0x7fdd1013f710 2024-03-25T18:29:58.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.417+0000 7fdd05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.417+0000 7fdd05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.417+0000 7fdd05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.417+0000 7fdd05ffb640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:29:58.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.417+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf40051d0 con 0x7fdd1013f710 2024-03-25T18:29:58.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.421+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654900172 0 0) 0x7fdd10137c00 con 0x7fdd100a98f0 2024-03-25T18:29:58.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.421+0000 7fdd05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.421+0000 7fdd05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.421+0000 7fdd05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.421+0000 7fdd05ffb640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:29:58.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.421+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd101379b0 con 0x7fdd100a98f0 2024-03-25T18:29:58.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.427+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139011094 0 0) 0x7fdcf4004eb0 con 0x7fdd101430c0 2024-03-25T18:29:58.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.428+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd10137c00 con 0x7fdd101430c0 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.428+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdd08001b30 con 0x7fdd101430c0 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.428+0000 7fdd05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.428+0000 7fdd05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.428+0000 7fdd05ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:58.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.428+0000 7fdd05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:58.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.433+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1365448019 0 0) 0x7fdd101379b0 con 0x7fdd100a98f0 2024-03-25T18:29:58.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.433+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcf4004eb0 con 0x7fdd100a98f0 2024-03-25T18:29:58.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.433+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdd0c001830 con 0x7fdd100a98f0 2024-03-25T18:29:58.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.433+0000 7fdd05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.433+0000 7fdd05ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:58.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.433+0000 7fdd05ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:58.434 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:58.435 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:58.435 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.435 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.435 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.435 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:58.435 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:58.435 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:58.435 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:58.435 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:58.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1168303431 0 0) 0x7fdd10137c00 con 0x7fdd101430c0 2024-03-25T18:29:58.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 >> v1:172.21.15.55:6790/0 conn(0x7fdd100a98f0 legacy=0x7fdd101db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 >> v1:172.21.15.55:6789/0 conn(0x7fdd1013f710 legacy=0x7fdd10138550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 1 monclient: found mon.b 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd10138c60 con 0x7fdd101430c0 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd1757b640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd1757b640 10 monclient: _renew_subs 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd1757b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd10137e70 con 0x7fdd101430c0 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd1757b640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:58.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd1757b640 10 monclient: _renew_subs 2024-03-25T18:29:58.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd1757b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:58.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd101e3520 con 0x7fdd101430c0 2024-03-25T18:29:58.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.437+0000 7fdd1757b640 1 librados: init done 2024-03-25T18:29:58.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.438+0000 7fdd1757b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:58.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.438+0000 7fdd1757b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:58.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.438+0000 7fdd1757b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:58.440 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.438+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd100abd80 con 0x7fdd101430c0 2024-03-25T18:29:58.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.702+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd08001fa0 con 0x7fdd101430c0 2024-03-25T18:29:58.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.702+0000 7fdd05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:58.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.702+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdd08004d00 con 0x7fdd101430c0 2024-03-25T18:29:58.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.702+0000 7fdd05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.702+0000 7fdd05ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:58.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.702+0000 7fdd05ffb640 10 monclient: dump: 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:29:58.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.702+0000 7fdd077fe640 4 set_mon_vals no callback set 2024-03-25T18:29:58.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.703+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fdd08004ee0 con 0x7fdd101430c0 2024-03-25T18:29:58.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.703+0000 7fdd15317640 10 monclient: get_auth_request con 0x7fdcf406e110 auth_method 0 2024-03-25T18:29:58.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.703+0000 7fdd05ffb640 10 client.4294.objecter ms_handle_connect 0x7fdcf406e110 2024-03-25T18:29:58.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.709+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fdd08054a10 con 0x7fdd101430c0 2024-03-25T18:29:58.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.709+0000 7fdd05ffb640 10 client.4294.objecter ms_dispatch 0x7fdd10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:58.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.709+0000 7fdd05ffb640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:58.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.709+0000 7fdd05ffb640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:58.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.709+0000 7fdd05ffb640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T18:29:58.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.709+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd08050520 con 0x7fdd101430c0 2024-03-25T18:29:58.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.709+0000 7fdd05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:58.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.709+0000 7fdd05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:58.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.936+0000 7fdd1757b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:58.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.936+0000 7fdd1757b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:58.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.936+0000 7fdd1757b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:29:58.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.936+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd100627b0 con 0x7fdd101430c0 2024-03-25T18:29:58.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.953+0000 7fdd05ffb640 1 -- 172.21.15.55:0/2554545434 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fdd08052450 con 0x7fdd101430c0 2024-03-25T18:29:58.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.953+0000 7fdd05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:58.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.953+0000 7fdd05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:58.954 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:29:58.954 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.955+0000 7fdd1757b640 10 librados: watch_flush enter 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.955+0000 7fdd1757b640 10 librados: watch_flush exit 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.955+0000 7fdd1757b640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.955+0000 7fdd1757b640 10 client.4294.objecter successfully canceled tick 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.955+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 >> v1:172.21.15.55:6800/286536064 conn(0x7fdcf406e110 legacy=0x7fdcf40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.955+0000 7fdd1757b640 10 monclient: shutdown 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.955+0000 7fdd1757b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.955+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 >> v1:172.21.15.133:6789/0 conn(0x7fdd101430c0 legacy=0x7fdd101e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.956+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 shutdown_connections 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.956+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 >> 172.21.15.55:0/2554545434 conn(0x7fdd100b7ea0 msgr2=0x7fdd10074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.956+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 shutdown_connections 2024-03-25T18:29:58.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.956+0000 7fdd1757b640 1 -- 172.21.15.55:0/2554545434 wait complete. 2024-03-25T18:29:58.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:29:58.956+0000 7fdd1757b640 1 librados: shutdown 2024-03-25T18:29:58.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:05.775 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:05.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.890+0000 7f6cb62f1640 10 monclient: get_monmap_and_config 2024-03-25T18:30:05.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.890+0000 7f6cb62f1640 10 monclient: build_initial_monmap 2024-03-25T18:30:05.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.890+0000 7f6cb62f1640 10 monclient: monmap: 2024-03-25T18:30:05.892 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:05.892 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:05.892 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:30:05.891418+0000 2024-03-25T18:30:05.892 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:30:05.891418+0000 2024-03-25T18:30:05.892 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:05.893 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:05.893 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:05.893 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:30:05.893 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:30:05.893 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:05.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.891+0000 7f6cb62f1640 1 Processor -- start 2024-03-25T18:30:05.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.891+0000 7f6cb62f1640 1 -- start start 2024-03-25T18:30:05.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.891+0000 7f6cb62f1640 10 monclient: init 2024-03-25T18:30:05.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:05.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:05.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb00a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:05.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:05.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb01430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:05.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb0142520 con 0x7f6cb013f710 2024-03-25T18:30:05.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00b28a0 con 0x7f6cb00a98f0 2024-03-25T18:30:05.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00b2a30 con 0x7f6cb01430c0 2024-03-25T18:30:05.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:05.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.892+0000 7f6cb62f1640 10 monclient(hunting): authenticate will time out at 7410.149482s 2024-03-25T18:30:05.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.904+0000 7f6caf7fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f6cb013f710 0x7f6cb01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54724/0 (socket says 172.21.15.55:54724) 2024-03-25T18:30:05.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.904+0000 7f6caf7fe640 1 -- 172.21.15.55:0/1098995393 learned_addr learned my addr 172.21.15.55:0/1098995393 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:05.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.911+0000 7f6caeffd640 1 -- 172.21.15.55:0/1098995393 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724926182 0 0) 0x7f6cb0142520 con 0x7f6cb013f710 2024-03-25T18:30:05.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.911+0000 7f6caeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.911+0000 7f6caeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.911+0000 7f6caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.911+0000 7f6caeffd640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:30:05.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.911+0000 7f6caeffd640 1 -- 172.21.15.55:0/1098995393 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c98003520 con 0x7f6cb013f710 2024-03-25T18:30:05.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 1 -- 172.21.15.55:0/1098995393 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4120840719 0 0) 0x7f6c98003520 con 0x7f6cb013f710 2024-03-25T18:30:05.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 1 -- 172.21.15.55:0/1098995393 >> v1:172.21.15.55:6790/0 conn(0x7f6cb01430c0 legacy=0x7f6cb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 1 -- 172.21.15.55:0/1098995393 >> v1:172.21.15.133:6789/0 conn(0x7f6cb00a98f0 legacy=0x7f6cb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:05.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 1 monclient: found mon.noname-a 2024-03-25T18:30:05.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:05.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:30:05.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 1 -- 172.21.15.55:0/1098995393 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb00b44c0 con 0x7f6cb013f710 2024-03-25T18:30:05.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:05.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:05.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:05.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 1 -- 172.21.15.55:0/1098995393 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6ca4001690 con 0x7f6cb013f710 2024-03-25T18:30:05.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:05.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6caeffd640 10 monclient: dump: 2024-03-25T18:30:05.927 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:05.927 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:05.927 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:05.927 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:05.927 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.928 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:05.928 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:05.928 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:05.928 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:05.928 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:05.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6cb62f1640 5 monclient: authenticate success, global_id 4314 2024-03-25T18:30:05.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.923+0000 7f6cb62f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:05.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.932+0000 7f6caeffd640 1 -- 172.21.15.55:0/1098995393 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca40017f0 con 0x7f6cb013f710 2024-03-25T18:30:05.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.932+0000 7f6caeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:05.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.932+0000 7f6cb62f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:05.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.932+0000 7f6cb62f1640 4 set_mon_vals no callback set 2024-03-25T18:30:05.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.933+0000 7f6cb62f1640 10 monclient: shutdown 2024-03-25T18:30:05.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.933+0000 7f6cb62f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:05.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.933+0000 7f6cb62f1640 1 -- 172.21.15.55:0/1098995393 >> v1:172.21.15.55:6789/0 conn(0x7f6cb013f710 legacy=0x7f6cb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.933+0000 7f6cb62f1640 1 -- 172.21.15.55:0/1098995393 shutdown_connections 2024-03-25T18:30:05.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.933+0000 7f6cb62f1640 1 -- 172.21.15.55:0/1098995393 >> 172.21.15.55:0/1098995393 conn(0x7f6cb00b7ea0 msgr2=0x7f6cb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:05.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.933+0000 7f6cb62f1640 1 -- 172.21.15.55:0/1098995393 shutdown_connections 2024-03-25T18:30:05.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.933+0000 7f6cb62f1640 1 -- 172.21.15.55:0/1098995393 wait complete. 2024-03-25T18:30:05.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.934+0000 7f6cb62f1640 10 monclient: build_initial_monmap 2024-03-25T18:30:05.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.934+0000 7f6cb62f1640 10 monclient: monmap: 2024-03-25T18:30:05.937 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:05.937 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:05.937 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:30:05.937 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:30:05.937 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:05.937 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:30:05.938 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:05.938 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:30:05.938 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:30:05.938 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:05.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.934+0000 7f6cb62f1640 1 librados: starting msgr at 2024-03-25T18:30:05.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.934+0000 7f6cb62f1640 1 librados: starting objecter 2024-03-25T18:30:05.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.935+0000 7f6cb62f1640 1 Processor -- start 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.935+0000 7f6cb62f1640 1 -- start start 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.935+0000 7f6cb62f1640 1 librados: setting wanted keys 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.935+0000 7f6cb62f1640 1 librados: calling monclient init 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.935+0000 7f6cb62f1640 10 monclient: init 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.935+0000 7f6cb62f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.935+0000 7f6cb62f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.936+0000 7f6cb62f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb00a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.936+0000 7f6cb62f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb013f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.936+0000 7f6cb62f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb01430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.936+0000 7f6cb62f1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb0139460 con 0x7f6cb00a98f0 2024-03-25T18:30:05.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.936+0000 7f6cb62f1640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb006ee90 con 0x7f6cb013f710 2024-03-25T18:30:05.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.936+0000 7f6cb62f1640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb006f170 con 0x7f6cb01430c0 2024-03-25T18:30:05.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.936+0000 7f6cb62f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:05.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.936+0000 7f6cb62f1640 10 monclient(hunting): authenticate will time out at 7410.193595s 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.941+0000 7f6caf7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f6cb013f710 0x7f6cb014e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:33270/0 (socket says 172.21.15.55:33270) 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.941+0000 7f6caf7fe640 1 -- 172.21.15.55:0/2567859967 learned_addr learned my addr 172.21.15.55:0/2567859967 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:05.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.942+0000 7f6cacff9640 10 client.?.objecter ms_handle_connect 0x7f6cb013f710 2024-03-25T18:30:05.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.942+0000 7f6cacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:05.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.955+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492760363 0 0) 0x7f6cb006ee90 con 0x7f6cb013f710 2024-03-25T18:30:05.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.955+0000 7f6cacff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.955+0000 7f6cacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.955+0000 7f6cacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.955+0000 7f6cacff9640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:30:05.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.955+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c004eb0 con 0x7f6cb013f710 2024-03-25T18:30:05.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.957+0000 7f6cacff9640 10 client.?.objecter ms_handle_connect 0x7f6cb00a98f0 2024-03-25T18:30:05.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.957+0000 7f6cacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:05.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.966+0000 7f6cacff9640 10 client.?.objecter ms_handle_connect 0x7f6cb01430c0 2024-03-25T18:30:05.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.966+0000 7f6cacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.967+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915132315 0 0) 0x7f6cb0139460 con 0x7f6cb00a98f0 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.967+0000 7f6cacff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.967+0000 7f6cacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.967+0000 7f6cacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.967+0000 7f6cacff9640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.967+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb006ee90 con 0x7f6cb00a98f0 2024-03-25T18:30:05.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.969+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773164728 0 0) 0x7f6c9c004eb0 con 0x7f6cb013f710 2024-03-25T18:30:05.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.969+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb0139460 con 0x7f6cb013f710 2024-03-25T18:30:05.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.969+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6ca4001ca0 con 0x7f6cb013f710 2024-03-25T18:30:05.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.969+0000 7f6cacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.969+0000 7f6cacff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:05.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.969+0000 7f6cacff9640 10 monclient(hunting): dump: 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:05.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.969+0000 7f6cacff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:05.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.978+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934071256 0 0) 0x7f6cb006f170 con 0x7f6cb01430c0 2024-03-25T18:30:05.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.979+0000 7f6cacff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.979+0000 7f6cacff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.979+0000 7f6cacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.979+0000 7f6cacff9640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.979+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c004eb0 con 0x7f6cb01430c0 2024-03-25T18:30:05.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.994+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 128888476 0 0) 0x7f6cb006ee90 con 0x7f6cb00a98f0 2024-03-25T18:30:05.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.994+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb006f170 con 0x7f6cb00a98f0 2024-03-25T18:30:05.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.994+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6ca00018b0 con 0x7f6cb00a98f0 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.994+0000 7f6cacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.994+0000 7f6cacff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.994+0000 7f6cacff9640 10 monclient(hunting): dump: 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1136409530 0 0) 0x7f6c9c004eb0 con 0x7f6cb01430c0 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb006ee90 con 0x7f6cb01430c0 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6ca8001a90 con 0x7f6cb01430c0 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 10 monclient(hunting): dump: 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:05.998 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1386812851 0 0) 0x7f6cb0139460 con 0x7f6cb013f710 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 >> v1:172.21.15.55:6790/0 conn(0x7f6cb01430c0 legacy=0x7f6cb01e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 >> v1:172.21.15.55:6789/0 conn(0x7f6cb00a98f0 legacy=0x7f6cb014d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 1 monclient: found mon.b 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb01390f0 con 0x7f6cb013f710 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cb62f1640 5 monclient: authenticate success, global_id 4300 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cb62f1640 10 monclient: _renew_subs 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cb62f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.996+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb006dea0 con 0x7f6cb013f710 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.997+0000 7f6cb62f1640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.997+0000 7f6cb62f1640 10 monclient: _renew_subs 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.997+0000 7f6cb62f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.997+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb00b7320 con 0x7f6cb013f710 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.997+0000 7f6cb62f1640 1 librados: init done 2024-03-25T18:30:06.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.998+0000 7f6cb62f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:06.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.998+0000 7f6cb62f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:06.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.998+0000 7f6cb62f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:06.001 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:05.998+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb00627b0 con 0x7f6cb013f710 2024-03-25T18:30:06.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.273+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca4001220 con 0x7f6cb013f710 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.273+0000 7f6cacff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.273+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6ca4004cf0 con 0x7f6cb013f710 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.273+0000 7f6cacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.273+0000 7f6cacff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.273+0000 7f6cacff9640 10 monclient: dump: 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:06.275 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:06.276 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:06.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.273+0000 7f6cae7fc640 4 set_mon_vals no callback set 2024-03-25T18:30:06.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.274+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f6ca4004ed0 con 0x7f6cb013f710 2024-03-25T18:30:06.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.275+0000 7f6caffff640 10 monclient: get_auth_request con 0x7f6c9c06e160 auth_method 0 2024-03-25T18:30:06.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.275+0000 7f6cacff9640 10 client.4300.objecter ms_handle_connect 0x7f6c9c06e160 2024-03-25T18:30:06.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.283+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f6ca40549b0 con 0x7f6cb013f710 2024-03-25T18:30:06.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.283+0000 7f6cacff9640 10 client.4300.objecter ms_dispatch 0x7f6cb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:06.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.283+0000 7f6cacff9640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:06.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.283+0000 7f6cacff9640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:06.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.283+0000 7f6cacff9640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T18:30:06.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.283+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ca40502f0 con 0x7f6cb013f710 2024-03-25T18:30:06.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.283+0000 7f6cacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:06.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.283+0000 7f6cacff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:06.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.511+0000 7f6cb62f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:06.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.511+0000 7f6cb62f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:06.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.511+0000 7f6cb62f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:06.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.511+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cb014d680 con 0x7f6cb013f710 2024-03-25T18:30:06.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.597+0000 7f6cacff9640 1 -- 172.21.15.55:0/2567859967 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f6cb014d680 con 0x7f6cb013f710 2024-03-25T18:30:06.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.597+0000 7f6cacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:06.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.597+0000 7f6cacff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:06.598 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:30:06.598 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:06.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 10 librados: watch_flush enter 2024-03-25T18:30:06.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 10 librados: watch_flush exit 2024-03-25T18:30:06.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T18:30:06.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 10 client.4300.objecter successfully canceled tick 2024-03-25T18:30:06.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 >> v1:172.21.15.55:6800/286536064 conn(0x7f6c9c06e160 legacy=0x7f6c9c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:06.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 10 monclient: shutdown 2024-03-25T18:30:06.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:06.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 >> v1:172.21.15.133:6789/0 conn(0x7f6cb013f710 legacy=0x7f6cb014e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:06.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 shutdown_connections 2024-03-25T18:30:06.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 >> 172.21.15.55:0/2567859967 conn(0x7f6cb00b7ea0 msgr2=0x7f6cb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:06.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 shutdown_connections 2024-03-25T18:30:06.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 1 -- 172.21.15.55:0/2567859967 wait complete. 2024-03-25T18:30:06.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:06.599+0000 7f6cb62f1640 1 librados: shutdown 2024-03-25T18:30:06.614 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:13.421 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:13.538 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.536+0000 7ff3f750e640 10 monclient: get_monmap_and_config 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.536+0000 7ff3f750e640 10 monclient: build_initial_monmap 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.537+0000 7ff3f750e640 10 monclient: monmap: 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:30:13.538841+0000 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:30:13.538841+0000 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.537+0000 7ff3f750e640 1 Processor -- start 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.537+0000 7ff3f750e640 1 -- start start 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.537+0000 7ff3f750e640 10 monclient: init 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f00b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f00b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f014be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f00af940 con 0x7ff3f00b1730 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f00b0a70 con 0x7ff3f014be90 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f00b0d50 con 0x7ff3f00b4d60 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.538+0000 7ff3f750e640 10 monclient(hunting): authenticate will time out at 7417.797058s 2024-03-25T18:30:13.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.544+0000 7ff3f5aab640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7ff3f014be90 0x7ff3f014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:33742/0 (socket says 172.21.15.55:33742) 2024-03-25T18:30:13.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.544+0000 7ff3f5aab640 1 -- 172.21.15.55:0/1145209306 learned_addr learned my addr 172.21.15.55:0/1145209306 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.555+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687125884 0 0) 0x7ff3f00b0a70 con 0x7ff3f014be90 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.555+0000 7ff3e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.555+0000 7ff3e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.555+0000 7ff3e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.555+0000 7ff3e7fff640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.555+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc003520 con 0x7ff3f014be90 2024-03-25T18:30:13.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.559+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628035579 0 0) 0x7ff3f00af940 con 0x7ff3f00b1730 2024-03-25T18:30:13.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.559+0000 7ff3e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.559+0000 7ff3e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.559+0000 7ff3e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.559+0000 7ff3e7fff640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:30:13.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.559+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f00b0a70 con 0x7ff3f00b1730 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694819400 0 0) 0x7ff3dc003520 con 0x7ff3f014be90 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 >> v1:172.21.15.55:6790/0 conn(0x7ff3f00b4d60 legacy=0x7ff3f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 >> v1:172.21.15.55:6789/0 conn(0x7ff3f00b1730 legacy=0x7ff3f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 1 monclient: found mon.noname-c 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f00b1030 con 0x7ff3f014be90 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 10 monclient: _finish_auth 0 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3e8001a10 con 0x7ff3f014be90 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3e7fff640 10 monclient: dump: 2024-03-25T18:30:13.563 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3f750e640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.560+0000 7ff3f750e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:13.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.582+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3e8001b70 con 0x7ff3f014be90 2024-03-25T18:30:13.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.582+0000 7ff3e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:13.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.582+0000 7ff3e7fff640 1 -- 172.21.15.55:0/1145209306 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3e8001f80 con 0x7ff3f014be90 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.582+0000 7ff3f750e640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.582+0000 7ff3f750e640 4 set_mon_vals no callback set 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.582+0000 7ff3e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.582+0000 7ff3e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.582+0000 7ff3e7fff640 10 monclient: dump: 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:13.585 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:13.586 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:13.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.583+0000 7ff3f750e640 10 monclient: shutdown 2024-03-25T18:30:13.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.583+0000 7ff3f750e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:13.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.583+0000 7ff3f750e640 1 -- 172.21.15.55:0/1145209306 >> v1:172.21.15.133:6789/0 conn(0x7ff3f014be90 legacy=0x7ff3f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.583+0000 7ff3f750e640 1 -- 172.21.15.55:0/1145209306 shutdown_connections 2024-03-25T18:30:13.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.583+0000 7ff3f750e640 1 -- 172.21.15.55:0/1145209306 >> 172.21.15.55:0/1145209306 conn(0x7ff3f00617a0 msgr2=0x7ff3f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:13.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.583+0000 7ff3f750e640 1 -- 172.21.15.55:0/1145209306 shutdown_connections 2024-03-25T18:30:13.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.583+0000 7ff3f750e640 1 -- 172.21.15.55:0/1145209306 wait complete. 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.584+0000 7ff3f750e640 10 monclient: build_initial_monmap 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.584+0000 7ff3f750e640 10 monclient: monmap: 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.585+0000 7ff3f750e640 1 librados: starting msgr at 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.585+0000 7ff3f750e640 1 librados: starting objecter 2024-03-25T18:30:13.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.585+0000 7ff3f750e640 1 Processor -- start 2024-03-25T18:30:13.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.585+0000 7ff3f750e640 1 -- start start 2024-03-25T18:30:13.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.585+0000 7ff3f750e640 1 librados: setting wanted keys 2024-03-25T18:30:13.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.585+0000 7ff3f750e640 1 librados: calling monclient init 2024-03-25T18:30:13.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.585+0000 7ff3f750e640 10 monclient: init 2024-03-25T18:30:13.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:13.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:13.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f00b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:13.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f00b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:13.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f014be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:13.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f01419e0 con 0x7ff3f014be90 2024-03-25T18:30:13.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f0141c30 con 0x7ff3f00b4d60 2024-03-25T18:30:13.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f0140810 con 0x7ff3f00b1730 2024-03-25T18:30:13.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:13.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.586+0000 7ff3f750e640 10 monclient(hunting): authenticate will time out at 7417.844794s 2024-03-25T18:30:13.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.594+0000 7ff3f4aa9640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7ff3f00b4d60 0x7ff3f01dad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:33768/0 (socket says 172.21.15.55:33768) 2024-03-25T18:30:13.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.594+0000 7ff3f4aa9640 1 -- 172.21.15.55:0/230314335 learned_addr learned my addr 172.21.15.55:0/230314335 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.600+0000 7ff3e5ffb640 10 client.?.objecter ms_handle_connect 0x7ff3f00b1730 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.600+0000 7ff3e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:13.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.601+0000 7ff3e5ffb640 10 client.?.objecter ms_handle_connect 0x7ff3f00b4d60 2024-03-25T18:30:13.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.601+0000 7ff3e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:13.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.612+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451034170 0 0) 0x7ff3f0140810 con 0x7ff3f00b1730 2024-03-25T18:30:13.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.612+0000 7ff3e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.612+0000 7ff3e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.612+0000 7ff3e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.612+0000 7ff3e5ffb640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:30:13.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.612+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d4004eb0 con 0x7ff3f00b1730 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.614+0000 7ff3e5ffb640 10 client.?.objecter ms_handle_connect 0x7ff3f014be90 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.614+0000 7ff3e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:13.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.620+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060373641 0 0) 0x7ff3f0141c30 con 0x7ff3f00b4d60 2024-03-25T18:30:13.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.620+0000 7ff3e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.620+0000 7ff3e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.620+0000 7ff3e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.620+0000 7ff3e5ffb640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:30:13.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.620+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f0140810 con 0x7ff3f00b4d60 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.630+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1527305858 0 0) 0x7ff3f0140810 con 0x7ff3f00b4d60 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.630+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f0141c30 con 0x7ff3f00b4d60 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.630+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3e00019f0 con 0x7ff3f00b4d60 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.630+0000 7ff3e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.630+0000 7ff3e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.630+0000 7ff3e5ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.632 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.633 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.633 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:13.633 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:13.633 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:13.633 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:13.633 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:13.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.630+0000 7ff3e5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:13.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.634+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804050942 0 0) 0x7ff3f01419e0 con 0x7ff3f014be90 2024-03-25T18:30:13.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.634+0000 7ff3e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.634+0000 7ff3e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.634+0000 7ff3e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.634+0000 7ff3e5ffb640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:30:13.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.634+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f0140810 con 0x7ff3f014be90 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.649+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1638628581 0 0) 0x7ff3d4004eb0 con 0x7ff3f00b1730 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.649+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f01419e0 con 0x7ff3f00b1730 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.649+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3ec0018c0 con 0x7ff3f00b1730 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.649+0000 7ff3e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.649+0000 7ff3e5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.649+0000 7ff3e5ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.651 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.652 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:13.652 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:13.652 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:13.652 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:13.652 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:13.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.651+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1337121731 0 0) 0x7ff3f0141c30 con 0x7ff3f00b4d60 2024-03-25T18:30:13.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.651+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 >> v1:172.21.15.55:6790/0 conn(0x7ff3f00b1730 legacy=0x7ff3f0131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.651+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 >> v1:172.21.15.55:6789/0 conn(0x7ff3f014be90 legacy=0x7ff3f01e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3e5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:13.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3e5ffb640 1 monclient: found mon.b 2024-03-25T18:30:13.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:13.654 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3e5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f0140380 con 0x7ff3f00b4d60 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3e5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3e5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3f750e640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3f750e640 10 monclient: _renew_subs 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3f750e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f01384b0 con 0x7ff3f00b4d60 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3f750e640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3f750e640 10 monclient: _renew_subs 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3f750e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f0132700 con 0x7ff3f00b4d60 2024-03-25T18:30:13.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.652+0000 7ff3f750e640 1 librados: init done 2024-03-25T18:30:13.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.653+0000 7ff3f750e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:13.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.653+0000 7ff3f750e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:13.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.653+0000 7ff3f750e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:13.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.653+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f00ae550 con 0x7ff3f00b4d60 2024-03-25T18:30:13.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.927+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3e00035f0 con 0x7ff3f00b4d60 2024-03-25T18:30:13.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.927+0000 7ff3e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:13.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.927+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff3e0003980 con 0x7ff3f00b4d60 2024-03-25T18:30:13.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.927+0000 7ff3e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.927+0000 7ff3e77fe640 4 set_mon_vals no callback set 2024-03-25T18:30:13.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.927+0000 7ff3e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:13.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.927+0000 7ff3e5ffb640 10 monclient: dump: 2024-03-25T18:30:13.930 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:13.930 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:13.930 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.930 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:13.930 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.931 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:13.931 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:13.931 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:13.931 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:13.931 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:13.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.928+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7ff3e0003ba0 con 0x7ff3f00b4d60 2024-03-25T18:30:13.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.929+0000 7ff3f52aa640 10 monclient: get_auth_request con 0x7ff3d406dfb0 auth_method 0 2024-03-25T18:30:13.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.929+0000 7ff3e5ffb640 10 client.4306.objecter ms_handle_connect 0x7ff3d406dfb0 2024-03-25T18:30:13.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.933+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7ff3e0054790 con 0x7ff3f00b4d60 2024-03-25T18:30:13.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.933+0000 7ff3e5ffb640 10 client.4306.objecter ms_dispatch 0x7ff3f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:13.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.933+0000 7ff3e5ffb640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:13.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.933+0000 7ff3e5ffb640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:13.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.933+0000 7ff3e5ffb640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T18:30:13.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.933+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3e00502a0 con 0x7ff3f00b4d60 2024-03-25T18:30:13.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.933+0000 7ff3e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:13.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:13.933+0000 7ff3e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:14.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.160+0000 7ff3f750e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:14.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.160+0000 7ff3f750e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:14.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.160+0000 7ff3f750e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:30:14.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.160+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3f01409a0 con 0x7ff3f00b4d60 2024-03-25T18:30:14.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.180+0000 7ff3e5ffb640 1 -- 172.21.15.55:0/230314335 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7ff3e00521d0 con 0x7ff3f00b4d60 2024-03-25T18:30:14.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.180+0000 7ff3e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:14.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.180+0000 7ff3e5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:14.182 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:30:14.182 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:14.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 10 librados: watch_flush enter 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 10 librados: watch_flush exit 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 10 client.4306.objecter successfully canceled tick 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 >> v1:172.21.15.55:6800/286536064 conn(0x7ff3d406dfb0 legacy=0x7ff3d4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 10 monclient: shutdown 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 >> v1:172.21.15.133:6789/0 conn(0x7ff3f00b4d60 legacy=0x7ff3f01dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 shutdown_connections 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.182+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 >> 172.21.15.55:0/230314335 conn(0x7ff3f00617a0 msgr2=0x7ff3f013c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.183+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 shutdown_connections 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.183+0000 7ff3f750e640 1 -- 172.21.15.55:0/230314335 wait complete. 2024-03-25T18:30:14.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:14.183+0000 7ff3f750e640 1 librados: shutdown 2024-03-25T18:30:14.196 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:14.362 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:02:24.532 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:14.413 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:02:24.583 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:14.428 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:02:24.598 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:14.465 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:24.602 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:14.467 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:02:24.638 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:14.471 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:24.608 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:14.500 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:02:24.670 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:14.514 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:24.652 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:14.529 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:02:24.699 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:21.004 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:21.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.099+0000 7f853a872640 10 monclient: get_monmap_and_config 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.099+0000 7f853a872640 10 monclient: build_initial_monmap 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.099+0000 7f853a872640 10 monclient: monmap: 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:30:21.100574+0000 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:30:21.100574+0000 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:30:21.101 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.100+0000 7f853a872640 1 Processor -- start 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.100+0000 7f853a872640 1 -- start start 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.100+0000 7f853a872640 10 monclient: init 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 10 monclient(hunting): picked mon.noname-a con 0x7f85340b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 10 monclient(hunting): picked mon.noname-c con 0x7f85340b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 10 monclient(hunting): picked mon.noname-b con 0x7f853414be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340af940 con 0x7f85340b4d60 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340b0a70 con 0x7f85340b1730 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85340b0d50 con 0x7f853414be90 2024-03-25T18:30:21.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:21.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.101+0000 7f853a872640 10 monclient(hunting): authenticate will time out at 7425.358977s 2024-03-25T18:30:21.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.111+0000 7f8533fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f85340b4d60 0x7f853413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:36084/0 (socket says 172.21.15.55:36084) 2024-03-25T18:30:21.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.111+0000 7f8533fff640 1 -- 172.21.15.55:0/3593032024 learned_addr learned my addr 172.21.15.55:0/3593032024 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:21.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.118+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240796826 0 0) 0x7f85340af940 con 0x7f85340b4d60 2024-03-25T18:30:21.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.118+0000 7f8532ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.118+0000 7f8532ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.118+0000 7f8532ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.118+0000 7f8532ffd640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:30:21.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.118+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f851c003520 con 0x7f85340b4d60 2024-03-25T18:30:21.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.133+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610148581 0 0) 0x7f85340b0a70 con 0x7f85340b1730 2024-03-25T18:30:21.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.133+0000 7f8532ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.133+0000 7f8532ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.133+0000 7f8532ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.133+0000 7f8532ffd640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:30:21.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.133+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85340af940 con 0x7f85340b1730 2024-03-25T18:30:21.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 245180236 0 0) 0x7f851c003520 con 0x7f85340b4d60 2024-03-25T18:30:21.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 >> v1:172.21.15.55:6790/0 conn(0x7f853414be90 legacy=0x7f853414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 >> v1:172.21.15.133:6789/0 conn(0x7f85340b1730 legacy=0x7f85340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:21.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 1 monclient: found mon.noname-a 2024-03-25T18:30:21.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85340b1030 con 0x7f85340b4d60 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8524001690 con 0x7f85340b4d60 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f853a872640 5 monclient: authenticate success, global_id 4326 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f853a872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f8532ffd640 10 monclient: dump: 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:21.144 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:21.145 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.145 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:21.145 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:21.145 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:21.145 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:21.145 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:21.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.142+0000 7f853a872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:21.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.173+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85240017f0 con 0x7f85340b4d60 2024-03-25T18:30:21.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.173+0000 7f8532ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:21.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.173+0000 7f8532ffd640 1 -- 172.21.15.55:0/3593032024 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8524001c00 con 0x7f85340b4d60 2024-03-25T18:30:21.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.173+0000 7f8532ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.173+0000 7f8532ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:21.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.173+0000 7f8532ffd640 10 monclient: dump: 2024-03-25T18:30:21.174 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:21.174 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:21.174 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:21.175 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:21.175 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.175 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:21.175 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:21.175 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:21.175 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:21.175 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:21.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.173+0000 7f853a872640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:21.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.173+0000 7f853a872640 4 set_mon_vals no callback set 2024-03-25T18:30:21.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.174+0000 7f853a872640 10 monclient: shutdown 2024-03-25T18:30:21.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.174+0000 7f853a872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:21.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.174+0000 7f853a872640 1 -- 172.21.15.55:0/3593032024 >> v1:172.21.15.55:6789/0 conn(0x7f85340b4d60 legacy=0x7f853413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.174+0000 7f853a872640 1 -- 172.21.15.55:0/3593032024 shutdown_connections 2024-03-25T18:30:21.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.174+0000 7f853a872640 1 -- 172.21.15.55:0/3593032024 >> 172.21.15.55:0/3593032024 conn(0x7f85340617a0 msgr2=0x7f8534061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:21.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.174+0000 7f853a872640 1 -- 172.21.15.55:0/3593032024 shutdown_connections 2024-03-25T18:30:21.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.174+0000 7f853a872640 1 -- 172.21.15.55:0/3593032024 wait complete. 2024-03-25T18:30:21.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.174+0000 7f853a872640 10 monclient: build_initial_monmap 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.174+0000 7f853a872640 10 monclient: monmap: 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.175+0000 7f853a872640 1 librados: starting msgr at 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.175+0000 7f853a872640 1 librados: starting objecter 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.175+0000 7f853a872640 1 Processor -- start 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.175+0000 7f853a872640 1 -- start start 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.175+0000 7f853a872640 1 librados: setting wanted keys 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.175+0000 7f853a872640 1 librados: calling monclient init 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.175+0000 7f853a872640 10 monclient: init 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 10 monclient(hunting): picked mon.noname-c con 0x7f85340b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 10 monclient(hunting): picked mon.noname-b con 0x7f85340b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 10 monclient(hunting): picked mon.noname-a con 0x7f853414be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534138210 con 0x7f853414be90 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85341375a0 con 0x7f85340b4d60 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8534138440 con 0x7f85340b1730 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.176+0000 7f853a872640 10 monclient(hunting): authenticate will time out at 7425.433898s 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.182+0000 7f8533fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f85340b1730 0x7f8534139320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:43084/0 (socket says 172.21.15.55:43084) 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.183+0000 7f8533fff640 1 -- 172.21.15.55:0/3903078436 learned_addr learned my addr 172.21.15.55:0/3903078436 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:21.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.185+0000 7f8530ff9640 10 client.?.objecter ms_handle_connect 0x7f85340b1730 2024-03-25T18:30:21.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.185+0000 7f8530ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:21.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.187+0000 7f8530ff9640 10 client.?.objecter ms_handle_connect 0x7f853414be90 2024-03-25T18:30:21.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.187+0000 7f8530ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:21.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.188+0000 7f8530ff9640 10 client.?.objecter ms_handle_connect 0x7f85340b4d60 2024-03-25T18:30:21.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.188+0000 7f8530ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.201+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114972349 0 0) 0x7f8534138440 con 0x7f85340b1730 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.201+0000 7f8530ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.201+0000 7f8530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.201+0000 7f8530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.201+0000 7f8530ff9640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.201+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8520004eb0 con 0x7f85340b1730 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.213+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 95762957 0 0) 0x7f8520004eb0 con 0x7f85340b1730 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.213+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8534138440 con 0x7f85340b1730 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.213+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8524001bc0 con 0x7f85340b1730 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.213+0000 7f8530ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.213+0000 7f8530ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.213+0000 7f8530ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:21.214 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:21.215 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.215 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:21.215 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:21.215 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:21.215 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:21.215 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:21.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.213+0000 7f8530ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:21.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.215+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563583198 0 0) 0x7f85341375a0 con 0x7f85340b4d60 2024-03-25T18:30:21.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.215+0000 7f8530ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.215+0000 7f8530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.215+0000 7f8530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.215+0000 7f8530ff9640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:30:21.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.215+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8520004eb0 con 0x7f85340b4d60 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.227+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4185370368 0 0) 0x7f8534138440 con 0x7f85340b1730 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.227+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 >> v1:172.21.15.133:6789/0 conn(0x7f85340b4d60 legacy=0x7f853412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.227+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 >> v1:172.21.15.55:6789/0 conn(0x7f853414be90 legacy=0x7f85341e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.227+0000 7f8530ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.227+0000 7f8530ff9640 1 monclient: found mon.c 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f8530ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f8530ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85341364b0 con 0x7f85340b1730 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f8530ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f8530ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f8530ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f853a872640 5 monclient: authenticate success, global_id 4325 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f853a872640 10 monclient: _renew_subs 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f853a872640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853413b470 con 0x7f85340b1730 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f853a872640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:21.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f853a872640 10 monclient: _renew_subs 2024-03-25T18:30:21.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f853a872640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:21.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8534136230 con 0x7f85340b1730 2024-03-25T18:30:21.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.228+0000 7f853a872640 1 librados: init done 2024-03-25T18:30:21.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.229+0000 7f853a872640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:21.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.229+0000 7f853a872640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:21.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.229+0000 7f853a872640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:21.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.229+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85340627b0 con 0x7f85340b1730 2024-03-25T18:30:21.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.549+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85240011f0 con 0x7f85340b1730 2024-03-25T18:30:21.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.549+0000 7f8530ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:21.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.549+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8524004c70 con 0x7f85340b1730 2024-03-25T18:30:21.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.549+0000 7f8530ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.549+0000 7f8530ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.549+0000 7f8530ff9640 10 monclient: dump: 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:21.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.549+0000 7f85327fc640 4 set_mon_vals no callback set 2024-03-25T18:30:21.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.550+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f8524004e90 con 0x7f85340b1730 2024-03-25T18:30:21.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.550+0000 7f85337fe640 10 monclient: get_auth_request con 0x7f852006dd40 auth_method 0 2024-03-25T18:30:21.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.550+0000 7f8530ff9640 10 client.4325.objecter ms_handle_connect 0x7f852006dd40 2024-03-25T18:30:21.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.556+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f8524054910 con 0x7f85340b1730 2024-03-25T18:30:21.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.556+0000 7f8530ff9640 10 client.4325.objecter ms_dispatch 0x7f8534000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:21.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.556+0000 7f8530ff9640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:21.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.556+0000 7f8530ff9640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:21.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.556+0000 7f8530ff9640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-25T18:30:21.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.556+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8524050250 con 0x7f85340b1730 2024-03-25T18:30:21.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.556+0000 7f8530ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:21.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.556+0000 7f8530ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:21.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.820+0000 7f853a872640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:21.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.820+0000 7f853a872640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:21.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.820+0000 7f853a872640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:21.821 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.820+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8534136640 con 0x7f85340b1730 2024-03-25T18:30:21.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.845+0000 7f8530ff9640 1 -- 172.21.15.55:0/3903078436 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f8524052350 con 0x7f85340b1730 2024-03-25T18:30:21.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.845+0000 7f8530ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:21.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.845+0000 7f8530ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:21.847 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:30:21.847 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:21.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 10 librados: watch_flush enter 2024-03-25T18:30:21.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 10 librados: watch_flush exit 2024-03-25T18:30:21.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-25T18:30:21.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 10 client.4325.objecter successfully canceled tick 2024-03-25T18:30:21.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 >> v1:172.21.15.55:6800/286536064 conn(0x7f852006dd40 legacy=0x7f8520070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 10 monclient: shutdown 2024-03-25T18:30:21.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:21.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 >> v1:172.21.15.55:6790/0 conn(0x7f85340b1730 legacy=0x7f8534139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 shutdown_connections 2024-03-25T18:30:21.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 >> 172.21.15.55:0/3903078436 conn(0x7f85340617a0 msgr2=0x7f853413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:21.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 shutdown_connections 2024-03-25T18:30:21.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 1 -- 172.21.15.55:0/3903078436 wait complete. 2024-03-25T18:30:21.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:21.848+0000 7f853a872640 1 librados: shutdown 2024-03-25T18:30:21.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:28.668 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:28.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.788+0000 7f2f5d8d9640 10 monclient: get_monmap_and_config 2024-03-25T18:30:28.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.788+0000 7f2f5d8d9640 10 monclient: build_initial_monmap 2024-03-25T18:30:28.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.788+0000 7f2f5d8d9640 10 monclient: monmap: 2024-03-25T18:30:28.789 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:28.790 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:28.790 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:30:28.789510+0000 2024-03-25T18:30:28.790 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:30:28.789510+0000 2024-03-25T18:30:28.790 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:28.790 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:28.790 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:28.790 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:30:28.790 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:30:28.790 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:28.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.789+0000 7f2f5d8d9640 1 Processor -- start 2024-03-25T18:30:28.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.789+0000 7f2f5d8d9640 1 -- start start 2024-03-25T18:30:28.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.789+0000 7f2f5d8d9640 10 monclient: init 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f580a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5813f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f581430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f58142520 con 0x7f2f5813f710 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580b28a0 con 0x7f2f581430c0 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580b2a30 con 0x7f2f580a98f0 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:28.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.790+0000 7f2f5d8d9640 10 monclient(hunting): authenticate will time out at 7433.047855s 2024-03-25T18:30:28.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.805+0000 7f2f577fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2f581430c0 0x7f2f5814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:51700/0 (socket says 172.21.15.55:51700) 2024-03-25T18:30:28.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.805+0000 7f2f577fe640 1 -- 172.21.15.55:0/676912838 learned_addr learned my addr 172.21.15.55:0/676912838 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:28.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.818+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153816296 0 0) 0x7f2f580b2a30 con 0x7f2f580a98f0 2024-03-25T18:30:28.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.818+0000 7f2f567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.818+0000 7f2f567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.818+0000 7f2f567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.818+0000 7f2f567fc640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:30:28.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.818+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f44003520 con 0x7f2f580a98f0 2024-03-25T18:30:28.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.822+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024255588 0 0) 0x7f2f580b28a0 con 0x7f2f581430c0 2024-03-25T18:30:28.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.822+0000 7f2f567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.822+0000 7f2f567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.822+0000 7f2f567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.822+0000 7f2f567fc640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:30:28.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.822+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f580b2a30 con 0x7f2f581430c0 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1874519620 0 0) 0x7f2f44003520 con 0x7f2f580a98f0 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 >> v1:172.21.15.133:6789/0 conn(0x7f2f581430c0 legacy=0x7f2f5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 >> v1:172.21.15.55:6789/0 conn(0x7f2f5813f710 legacy=0x7f2f581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 1 monclient: found mon.noname-b 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580b44c0 con 0x7f2f580a98f0 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:28.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2f48001690 con 0x7f2f580a98f0 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f5d8d9640 5 monclient: authenticate success, global_id 4328 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f5d8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f567fc640 10 monclient: dump: 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:28.833 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:28.834 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:28.834 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.831+0000 7f2f5d8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:28.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.835+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f480017f0 con 0x7f2f580a98f0 2024-03-25T18:30:28.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.835+0000 7f2f567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:28.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.835+0000 7f2f567fc640 1 -- 172.21.15.55:0/676912838 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2f48001c00 con 0x7f2f580a98f0 2024-03-25T18:30:28.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.835+0000 7f2f567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f567fc640 10 monclient: dump: 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f5d8d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:28.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f5d8d9640 4 set_mon_vals no callback set 2024-03-25T18:30:28.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f5d8d9640 10 monclient: shutdown 2024-03-25T18:30:28.839 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f5d8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/676912838 >> v1:172.21.15.55:6790/0 conn(0x7f2f580a98f0 legacy=0x7f2f580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/676912838 shutdown_connections 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/676912838 >> 172.21.15.55:0/676912838 conn(0x7f2f580b7ea0 msgr2=0x7f2f580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/676912838 shutdown_connections 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.836+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/676912838 wait complete. 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.837+0000 7f2f5d8d9640 10 monclient: build_initial_monmap 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.837+0000 7f2f5d8d9640 10 monclient: monmap: 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:30:28.840 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.837+0000 7f2f5d8d9640 1 librados: starting msgr at 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.837+0000 7f2f5d8d9640 1 librados: starting objecter 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 1 Processor -- start 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 1 -- start start 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 1 librados: setting wanted keys 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 1 librados: calling monclient init 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 10 monclient: init 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f580a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f5813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:28.841 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f581430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:28.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f581db7d0 con 0x7f2f580a98f0 2024-03-25T18:30:28.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580b1420 con 0x7f2f5813f710 2024-03-25T18:30:28.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580b15b0 con 0x7f2f581430c0 2024-03-25T18:30:28.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:28.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.838+0000 7f2f5d8d9640 10 monclient(hunting): authenticate will time out at 7433.096340s 2024-03-25T18:30:28.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.850+0000 7f2f577fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f2f581430c0 0x7f2f581d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:45888/0 (socket says 172.21.15.55:45888) 2024-03-25T18:30:28.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.850+0000 7f2f577fe640 1 -- 172.21.15.55:0/1023211161 learned_addr learned my addr 172.21.15.55:0/1023211161 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:28.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.853+0000 7f2f54ff9640 10 client.?.objecter ms_handle_connect 0x7f2f581430c0 2024-03-25T18:30:28.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.853+0000 7f2f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:28.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.857+0000 7f2f54ff9640 10 client.?.objecter ms_handle_connect 0x7f2f5813f710 2024-03-25T18:30:28.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.857+0000 7f2f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:28.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.859+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745129044 0 0) 0x7f2f580b15b0 con 0x7f2f581430c0 2024-03-25T18:30:28.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.859+0000 7f2f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.859+0000 7f2f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.859+0000 7f2f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.859+0000 7f2f54ff9640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:30:28.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.859+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f38004eb0 con 0x7f2f581430c0 2024-03-25T18:30:28.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.860+0000 7f2f54ff9640 10 client.?.objecter ms_handle_connect 0x7f2f580a98f0 2024-03-25T18:30:28.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.860+0000 7f2f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:28.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.872+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851785469 0 0) 0x7f2f580b1420 con 0x7f2f5813f710 2024-03-25T18:30:28.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.872+0000 7f2f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.872+0000 7f2f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.872+0000 7f2f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.872+0000 7f2f54ff9640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:30:28.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.872+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f580b15b0 con 0x7f2f5813f710 2024-03-25T18:30:28.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.873+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813094960 0 0) 0x7f2f581db7d0 con 0x7f2f580a98f0 2024-03-25T18:30:28.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.873+0000 7f2f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.873+0000 7f2f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.873+0000 7f2f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.873+0000 7f2f54ff9640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:30:28.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.873+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f580b1420 con 0x7f2f580a98f0 2024-03-25T18:30:28.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.881+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1992062243 0 0) 0x7f2f38004eb0 con 0x7f2f581430c0 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.881+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f581db7d0 con 0x7f2f581430c0 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.881+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2f50001c10 con 0x7f2f581430c0 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.881+0000 7f2f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.881+0000 7f2f54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.881+0000 7f2f54ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:28.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.881+0000 7f2f54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.896+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2806619065 0 0) 0x7f2f580b15b0 con 0x7f2f5813f710 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.896+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f38004eb0 con 0x7f2f5813f710 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.896+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2f400018b0 con 0x7f2f5813f710 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.896+0000 7f2f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.896+0000 7f2f54ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.896+0000 7f2f54ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:28.897 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:28.898 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:28.898 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.898 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.898 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.898 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:28.898 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:28.898 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:28.898 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:28.898 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:28.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.898+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 380897659 0 0) 0x7f2f580b1420 con 0x7f2f580a98f0 2024-03-25T18:30:28.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.898+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f580b15b0 con 0x7f2f580a98f0 2024-03-25T18:30:28.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.898+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2f48001b00 con 0x7f2f580a98f0 2024-03-25T18:30:28.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.898+0000 7f2f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.898+0000 7f2f54ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.898+0000 7f2f54ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:28.900 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:28.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4149695224 0 0) 0x7f2f581db7d0 con 0x7f2f581430c0 2024-03-25T18:30:28.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 >> v1:172.21.15.133:6789/0 conn(0x7f2f5813f710 legacy=0x7f2f580b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 >> v1:172.21.15.55:6789/0 conn(0x7f2f580a98f0 legacy=0x7f2f581daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 1 monclient: found mon.c 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580b1740 con 0x7f2f581430c0 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f5d8d9640 5 monclient: authenticate success, global_id 4331 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f5d8d9640 10 monclient: _renew_subs 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f5d8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f581db1a0 con 0x7f2f581430c0 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f5d8d9640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f5d8d9640 10 monclient: _renew_subs 2024-03-25T18:30:28.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f5d8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:28.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f580b08f0 con 0x7f2f581430c0 2024-03-25T18:30:28.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.908+0000 7f2f5d8d9640 1 librados: init done 2024-03-25T18:30:28.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.909+0000 7f2f5d8d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:28.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.909+0000 7f2f5d8d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:28.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.909+0000 7f2f5d8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:28.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.909+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f580627b0 con 0x7f2f581430c0 2024-03-25T18:30:28.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.929+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f50004b60 con 0x7f2f581430c0 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.929+0000 7f2f54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.929+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2f50004e70 con 0x7f2f581430c0 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.929+0000 7f2f54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.929+0000 7f2f54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.929+0000 7f2f54ff9640 10 monclient: dump: 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:28.931 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:28.932 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:28.932 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:28.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.929+0000 7f2f55ffb640 4 set_mon_vals no callback set 2024-03-25T18:30:28.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.930+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f2f5001ae40 con 0x7f2f581430c0 2024-03-25T18:30:28.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.931+0000 7f2f56ffd640 10 monclient: get_auth_request con 0x7f2f3806e0d0 auth_method 0 2024-03-25T18:30:28.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.931+0000 7f2f54ff9640 10 client.4331.objecter ms_handle_connect 0x7f2f3806e0d0 2024-03-25T18:30:28.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.933+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f2f50054a90 con 0x7f2f581430c0 2024-03-25T18:30:28.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.933+0000 7f2f54ff9640 10 client.4331.objecter ms_dispatch 0x7f2f58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:28.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.933+0000 7f2f54ff9640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:28.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.933+0000 7f2f54ff9640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:28.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:28.933+0000 7f2f54ff9640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-25T18:30:29.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.222+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f500505a0 con 0x7f2f581430c0 2024-03-25T18:30:29.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.222+0000 7f2f54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:29.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.222+0000 7f2f54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:29.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.483+0000 7f2f5d8d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:29.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.483+0000 7f2f5d8d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:29.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.483+0000 7f2f5d8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:29.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.483+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f580abae0 con 0x7f2f581430c0 2024-03-25T18:30:29.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.520+0000 7f2f54ff9640 1 -- 172.21.15.55:0/1023211161 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f2f500524d0 con 0x7f2f581430c0 2024-03-25T18:30:29.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.520+0000 7f2f54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:29.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.520+0000 7f2f54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:29.521 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:30:29.521 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.522+0000 7f2f5d8d9640 10 librados: watch_flush enter 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.522+0000 7f2f5d8d9640 10 librados: watch_flush exit 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.522+0000 7f2f5d8d9640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.522+0000 7f2f5d8d9640 10 client.4331.objecter successfully canceled tick 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.522+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 >> v1:172.21.15.55:6800/286536064 conn(0x7f2f3806e0d0 legacy=0x7f2f38070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.522+0000 7f2f5d8d9640 10 monclient: shutdown 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.522+0000 7f2f5d8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.522+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 >> v1:172.21.15.55:6790/0 conn(0x7f2f581430c0 legacy=0x7f2f581d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.522+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 shutdown_connections 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.523+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 >> 172.21.15.55:0/1023211161 conn(0x7f2f580b7ea0 msgr2=0x7f2f5813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:29.524 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.523+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 shutdown_connections 2024-03-25T18:30:29.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.523+0000 7f2f5d8d9640 1 -- 172.21.15.55:0/1023211161 wait complete. 2024-03-25T18:30:29.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:29.523+0000 7f2f5d8d9640 1 librados: shutdown 2024-03-25T18:30:29.535 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:34.496 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:02:44.666 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:34.503 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:02:44.673 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:34.534 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:02:44.704 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:34.551 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:44.688 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:34.560 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:44.697 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:34.576 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:02:44.746 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:34.584 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:02:44.755 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:34.598 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:02:44.735 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:34.615 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:02:44.786 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:36.341 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:36.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.451+0000 7fdb76bc8640 10 monclient: get_monmap_and_config 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.451+0000 7fdb76bc8640 10 monclient: build_initial_monmap 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.452+0000 7fdb76bc8640 10 monclient: monmap: 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:30:36.452841+0000 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:30:36.452841+0000 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:30:36.453 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:36.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.452+0000 7fdb76bc8640 1 Processor -- start 2024-03-25T18:30:36.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.453+0000 7fdb76bc8640 1 -- start start 2024-03-25T18:30:36.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.453+0000 7fdb76bc8640 10 monclient: init 2024-03-25T18:30:36.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.453+0000 7fdb76bc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:36.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.453+0000 7fdb76bc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:36.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.453+0000 7fdb76bc8640 10 monclient(hunting): picked mon.noname-c con 0x7fdb700705b0 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:36.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.454+0000 7fdb76bc8640 10 monclient(hunting): picked mon.noname-a con 0x7fdb701402d0 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:36.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.454+0000 7fdb76bc8640 10 monclient(hunting): picked mon.noname-b con 0x7fdb70149ae0 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:36.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.454+0000 7fdb76bc8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb700718c0 con 0x7fdb701402d0 2024-03-25T18:30:36.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.454+0000 7fdb76bc8640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb700714c0 con 0x7fdb700705b0 2024-03-25T18:30:36.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.454+0000 7fdb76bc8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb70072b50 con 0x7fdb70149ae0 2024-03-25T18:30:36.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.454+0000 7fdb76bc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:36.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.454+0000 7fdb76bc8640 10 monclient(hunting): authenticate will time out at 7440.711618s 2024-03-25T18:30:36.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.464+0000 7fdb74964640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fdb700705b0 0x7fdb7013d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:56026/0 (socket says 172.21.15.55:56026) 2024-03-25T18:30:36.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.464+0000 7fdb74964640 1 -- 172.21.15.55:0/712475012 learned_addr learned my addr 172.21.15.55:0/712475012 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:36.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.481+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648926792 0 0) 0x7fdb700714c0 con 0x7fdb700705b0 2024-03-25T18:30:36.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.481+0000 7fdb6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.481+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.481+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.481+0000 7fdb6f7fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:30:36.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.481+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb60003520 con 0x7fdb700705b0 2024-03-25T18:30:36.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.483+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45330052 0 0) 0x7fdb700718c0 con 0x7fdb701402d0 2024-03-25T18:30:36.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.483+0000 7fdb6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.483+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.483+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.483+0000 7fdb6f7fe640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:30:36.484 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.483+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb700714c0 con 0x7fdb701402d0 2024-03-25T18:30:36.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.492+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657205062 0 0) 0x7fdb70072b50 con 0x7fdb70149ae0 2024-03-25T18:30:36.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.492+0000 7fdb6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.492+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.492+0000 7fdb6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.492+0000 7fdb6f7fe640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:30:36.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.492+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb700718c0 con 0x7fdb70149ae0 2024-03-25T18:30:36.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413335673 0 0) 0x7fdb700714c0 con 0x7fdb701402d0 2024-03-25T18:30:36.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 >> v1:172.21.15.55:6790/0 conn(0x7fdb70149ae0 legacy=0x7fdb7014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 >> v1:172.21.15.133:6789/0 conn(0x7fdb700705b0 legacy=0x7fdb7013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7013dd10 con 0x7fdb701402d0 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdb5c001670 con 0x7fdb701402d0 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb76bc8640 5 monclient: authenticate success, global_id 4338 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb76bc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb6f7fe640 10 monclient: dump: 2024-03-25T18:30:36.496 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:36.497 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.494+0000 7fdb76bc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:36.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.511+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb5c0017d0 con 0x7fdb701402d0 2024-03-25T18:30:36.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.511+0000 7fdb6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:36.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.511+0000 7fdb6f7fe640 1 -- 172.21.15.55:0/712475012 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdb5c001be0 con 0x7fdb701402d0 2024-03-25T18:30:36.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.511+0000 7fdb76bc8640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:36.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.511+0000 7fdb76bc8640 4 set_mon_vals no callback set 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.511+0000 7fdb6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.511+0000 7fdb6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.511+0000 7fdb6f7fe640 10 monclient: dump: 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:36.513 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.512+0000 7fdb76bc8640 10 monclient: shutdown 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.512+0000 7fdb76bc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.512+0000 7fdb76bc8640 1 -- 172.21.15.55:0/712475012 >> v1:172.21.15.55:6789/0 conn(0x7fdb701402d0 legacy=0x7fdb70140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.512+0000 7fdb76bc8640 1 -- 172.21.15.55:0/712475012 shutdown_connections 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.512+0000 7fdb76bc8640 1 -- 172.21.15.55:0/712475012 >> 172.21.15.55:0/712475012 conn(0x7fdb700b5d10 msgr2=0x7fdb700b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.512+0000 7fdb76bc8640 1 -- 172.21.15.55:0/712475012 shutdown_connections 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.512+0000 7fdb76bc8640 1 -- 172.21.15.55:0/712475012 wait complete. 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.513+0000 7fdb76bc8640 10 monclient: build_initial_monmap 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.513+0000 7fdb76bc8640 10 monclient: monmap: 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:30:36.515 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.513+0000 7fdb76bc8640 1 librados: starting msgr at 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.513+0000 7fdb76bc8640 1 librados: starting objecter 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.514+0000 7fdb76bc8640 1 Processor -- start 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.514+0000 7fdb76bc8640 1 -- start start 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.514+0000 7fdb76bc8640 1 librados: setting wanted keys 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.514+0000 7fdb76bc8640 1 librados: calling monclient init 2024-03-25T18:30:36.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.514+0000 7fdb76bc8640 10 monclient: init 2024-03-25T18:30:36.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:36.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:36.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 10 monclient(hunting): picked mon.noname-a con 0x7fdb700705b0 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:36.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 10 monclient(hunting): picked mon.noname-b con 0x7fdb701402d0 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:36.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 10 monclient(hunting): picked mon.noname-c con 0x7fdb70149ae0 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:36.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7006d960 con 0x7fdb700705b0 2024-03-25T18:30:36.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb70141550 con 0x7fdb701402d0 2024-03-25T18:30:36.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb701416e0 con 0x7fdb70149ae0 2024-03-25T18:30:36.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:36.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.515+0000 7fdb76bc8640 10 monclient(hunting): authenticate will time out at 7440.772612s 2024-03-25T18:30:36.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.530+0000 7fdb74964640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fdb700705b0 0x7fdb70141ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45612/0 (socket says 172.21.15.55:45612) 2024-03-25T18:30:36.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.530+0000 7fdb74964640 1 -- 172.21.15.55:0/997588265 learned_addr learned my addr 172.21.15.55:0/997588265 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:36.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.531+0000 7fdb6d7fa640 10 client.?.objecter ms_handle_connect 0x7fdb700705b0 2024-03-25T18:30:36.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.531+0000 7fdb6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:36.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.534+0000 7fdb6d7fa640 10 client.?.objecter ms_handle_connect 0x7fdb70149ae0 2024-03-25T18:30:36.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.534+0000 7fdb6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:36.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.540+0000 7fdb6d7fa640 10 client.?.objecter ms_handle_connect 0x7fdb701402d0 2024-03-25T18:30:36.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.540+0000 7fdb6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:36.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333024313 0 0) 0x7fdb701416e0 con 0x7fdb70149ae0 2024-03-25T18:30:36.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:30:36.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb54004eb0 con 0x7fdb70149ae0 2024-03-25T18:30:36.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272862816 0 0) 0x7fdb7006d960 con 0x7fdb700705b0 2024-03-25T18:30:36.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:30:36.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.544+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb701416e0 con 0x7fdb700705b0 2024-03-25T18:30:36.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.555+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539994740 0 0) 0x7fdb70141550 con 0x7fdb701402d0 2024-03-25T18:30:36.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.555+0000 7fdb6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.555+0000 7fdb6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.555+0000 7fdb6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.555+0000 7fdb6d7fa640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:30:36.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.555+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7006d960 con 0x7fdb701402d0 2024-03-25T18:30:36.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.557+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2878846470 0 0) 0x7fdb701416e0 con 0x7fdb700705b0 2024-03-25T18:30:36.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.557+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb70141550 con 0x7fdb700705b0 2024-03-25T18:30:36.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.557+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdb640018d0 con 0x7fdb700705b0 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.557+0000 7fdb6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.557+0000 7fdb6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.557+0000 7fdb6d7fa640 10 monclient(hunting): dump: 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:36.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.557+0000 7fdb6d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:36.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.559+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1208839131 0 0) 0x7fdb54004eb0 con 0x7fdb70149ae0 2024-03-25T18:30:36.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.559+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb701416e0 con 0x7fdb70149ae0 2024-03-25T18:30:36.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.559+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdb68001bd0 con 0x7fdb70149ae0 2024-03-25T18:30:36.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.559+0000 7fdb6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.559+0000 7fdb6d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.559+0000 7fdb6d7fa640 10 monclient(hunting): dump: 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2779912846 0 0) 0x7fdb701416e0 con 0x7fdb70149ae0 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 >> v1:172.21.15.133:6789/0 conn(0x7fdb701402d0 legacy=0x7fdb701438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 >> v1:172.21.15.55:6789/0 conn(0x7fdb700705b0 legacy=0x7fdb70141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 1 monclient: found mon.c 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7013f5e0 con 0x7fdb70149ae0 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:30:36.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb76bc8640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb76bc8640 10 monclient: _renew_subs 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb76bc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7013f830 con 0x7fdb70149ae0 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb76bc8640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb76bc8640 10 monclient: _renew_subs 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb76bc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb701e3610 con 0x7fdb70149ae0 2024-03-25T18:30:36.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.570+0000 7fdb76bc8640 1 librados: init done 2024-03-25T18:30:36.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.571+0000 7fdb76bc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:36.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.571+0000 7fdb76bc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:36.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.571+0000 7fdb76bc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:36.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.571+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb700627b0 con 0x7fdb70149ae0 2024-03-25T18:30:36.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.880+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb68002060 con 0x7fdb70149ae0 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.880+0000 7fdb6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.880+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdb68003bb0 con 0x7fdb70149ae0 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.880+0000 7fdb6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.880+0000 7fdb6d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.880+0000 7fdb6d7fa640 10 monclient: dump: 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:36.882 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:36.883 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:36.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.880+0000 7fdb6effd640 4 set_mon_vals no callback set 2024-03-25T18:30:36.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.881+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fdb68019b80 con 0x7fdb70149ae0 2024-03-25T18:30:36.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.882+0000 7fdb74964640 10 monclient: get_auth_request con 0x7fdb5406df70 auth_method 0 2024-03-25T18:30:36.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.882+0000 7fdb6d7fa640 10 client.4337.objecter ms_handle_connect 0x7fdb5406df70 2024-03-25T18:30:36.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.889+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fdb680538c0 con 0x7fdb70149ae0 2024-03-25T18:30:36.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.889+0000 7fdb6d7fa640 10 client.4337.objecter ms_dispatch 0x7fdb70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:36.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.889+0000 7fdb6d7fa640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:36.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.889+0000 7fdb6d7fa640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:36.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.889+0000 7fdb6d7fa640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T18:30:36.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.889+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb6804f200 con 0x7fdb70149ae0 2024-03-25T18:30:36.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.889+0000 7fdb6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:36.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:36.889+0000 7fdb6d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:37.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.124+0000 7fdb76bc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:37.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.124+0000 7fdb76bc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:37.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.124+0000 7fdb76bc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:37.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.124+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb700b4790 con 0x7fdb70149ae0 2024-03-25T18:30:37.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.218+0000 7fdb6d7fa640 1 -- 172.21.15.55:0/997588265 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fdb68051300 con 0x7fdb70149ae0 2024-03-25T18:30:37.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.219+0000 7fdb6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:37.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.219+0000 7fdb6d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:37.220 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:30:37.220 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:37.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 10 librados: watch_flush enter 2024-03-25T18:30:37.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 10 librados: watch_flush exit 2024-03-25T18:30:37.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T18:30:37.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 10 client.4337.objecter successfully canceled tick 2024-03-25T18:30:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 >> v1:172.21.15.55:6800/286536064 conn(0x7fdb5406df70 legacy=0x7fdb54070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 10 monclient: shutdown 2024-03-25T18:30:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 >> v1:172.21.15.55:6790/0 conn(0x7fdb70149ae0 legacy=0x7fdb70143fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 shutdown_connections 2024-03-25T18:30:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 >> 172.21.15.55:0/997588265 conn(0x7fdb700b5d10 msgr2=0x7fdb700b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 shutdown_connections 2024-03-25T18:30:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 1 -- 172.21.15.55:0/997588265 wait complete. 2024-03-25T18:30:37.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:37.221+0000 7fdb76bc8640 1 librados: shutdown 2024-03-25T18:30:37.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:44.039 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.153+0000 7fcaba488640 10 monclient: get_monmap_and_config 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.153+0000 7fcaba488640 10 monclient: build_initial_monmap 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.153+0000 7fcaba488640 10 monclient: monmap: 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:30:44.154162+0000 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:30:44.154162+0000 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:44.154 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:44.155 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:30:44.155 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:30:44.155 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:44.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 1 Processor -- start 2024-03-25T18:30:44.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 1 -- start start 2024-03-25T18:30:44.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 10 monclient: init 2024-03-25T18:30:44.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:44.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:44.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 10 monclient(hunting): picked mon.noname-a con 0x7fcab40b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:44.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 10 monclient(hunting): picked mon.noname-b con 0x7fcab40b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:44.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 10 monclient(hunting): picked mon.noname-c con 0x7fcab414be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:44.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40af940 con 0x7fcab40b4d60 2024-03-25T18:30:44.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40b0a70 con 0x7fcab414be90 2024-03-25T18:30:44.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40b0d50 con 0x7fcab40b1730 2024-03-25T18:30:44.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:44.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.154+0000 7fcaba488640 10 monclient(hunting): authenticate will time out at 7448.412295s 2024-03-25T18:30:44.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.162+0000 7fcab8a25640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fcab414be90 0x7fcab414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:52784/0 (socket says 172.21.15.55:52784) 2024-03-25T18:30:44.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.162+0000 7fcab8a25640 1 -- 172.21.15.55:0/3244518303 learned_addr learned my addr 172.21.15.55:0/3244518303 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:44.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.181+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986092917 0 0) 0x7fcab40b0d50 con 0x7fcab40b1730 2024-03-25T18:30:44.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.181+0000 7fcab2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.181+0000 7fcab2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.181+0000 7fcab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.181+0000 7fcab2ffd640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:30:44.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.181+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca9c003520 con 0x7fcab40b1730 2024-03-25T18:30:44.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.186+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812848450 0 0) 0x7fcab40b0a70 con 0x7fcab414be90 2024-03-25T18:30:44.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.186+0000 7fcab2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.186+0000 7fcab2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.186+0000 7fcab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.186+0000 7fcab2ffd640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:30:44.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.186+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab40b0d50 con 0x7fcab414be90 2024-03-25T18:30:44.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.188+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096373662 0 0) 0x7fcab40af940 con 0x7fcab40b4d60 2024-03-25T18:30:44.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.188+0000 7fcab2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.188+0000 7fcab2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.188+0000 7fcab2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.188+0000 7fcab2ffd640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:30:44.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.188+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab40b0a70 con 0x7fcab40b4d60 2024-03-25T18:30:44.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3067192772 0 0) 0x7fcab40b0a70 con 0x7fcab40b4d60 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 >> v1:172.21.15.55:6790/0 conn(0x7fcab40b1730 legacy=0x7fcab40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 >> v1:172.21.15.133:6789/0 conn(0x7fcab414be90 legacy=0x7fcab414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40b1030 con 0x7fcab40b4d60 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcaa40015b0 con 0x7fcab40b4d60 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcab2ffd640 10 monclient: dump: 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:44.201 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcaba488640 5 monclient: authenticate success, global_id 4344 2024-03-25T18:30:44.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.199+0000 7fcaba488640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:44.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.214+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaa4001710 con 0x7fcab40b4d60 2024-03-25T18:30:44.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcab2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcab2ffd640 1 -- 172.21.15.55:0/3244518303 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcaa4001b20 con 0x7fcab40b4d60 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcab2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcab2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcab2ffd640 10 monclient: dump: 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:44.216 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcaba488640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:44.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcaba488640 4 set_mon_vals no callback set 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcaba488640 10 monclient: shutdown 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcaba488640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcaba488640 1 -- 172.21.15.55:0/3244518303 >> v1:172.21.15.55:6789/0 conn(0x7fcab40b4d60 legacy=0x7fcab413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcaba488640 1 -- 172.21.15.55:0/3244518303 shutdown_connections 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcaba488640 1 -- 172.21.15.55:0/3244518303 >> 172.21.15.55:0/3244518303 conn(0x7fcab40617a0 msgr2=0x7fcab4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcaba488640 1 -- 172.21.15.55:0/3244518303 shutdown_connections 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.215+0000 7fcaba488640 1 -- 172.21.15.55:0/3244518303 wait complete. 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.216+0000 7fcaba488640 10 monclient: build_initial_monmap 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.216+0000 7fcaba488640 10 monclient: monmap: 2024-03-25T18:30:44.219 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 1 librados: starting msgr at 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 1 librados: starting objecter 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 1 Processor -- start 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 1 -- start start 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 1 librados: setting wanted keys 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 1 librados: calling monclient init 2024-03-25T18:30:44.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 10 monclient: init 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 10 monclient(hunting): picked mon.noname-c con 0x7fcab40b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.217+0000 7fcaba488640 10 monclient(hunting): picked mon.noname-a con 0x7fcab40b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.218+0000 7fcaba488640 10 monclient(hunting): picked mon.noname-b con 0x7fcab414be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.218+0000 7fcaba488640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab4140930 con 0x7fcab40b4d60 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.218+0000 7fcaba488640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab4141870 con 0x7fcab414be90 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.218+0000 7fcaba488640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab41327e0 con 0x7fcab40b1730 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.218+0000 7fcaba488640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:44.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.218+0000 7fcaba488640 10 monclient(hunting): authenticate will time out at 7448.475453s 2024-03-25T18:30:44.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.229+0000 7fcab37fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fcab40b4d60 0x7fcab41319c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:32886/0 (socket says 172.21.15.55:32886) 2024-03-25T18:30:44.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.229+0000 7fcab37fe640 1 -- 172.21.15.55:0/1419819272 learned_addr learned my addr 172.21.15.55:0/1419819272 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:44.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.230+0000 7fcab0ff9640 10 client.?.objecter ms_handle_connect 0x7fcab40b4d60 2024-03-25T18:30:44.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.230+0000 7fcab0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:44.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.234+0000 7fcab0ff9640 10 client.?.objecter ms_handle_connect 0x7fcab414be90 2024-03-25T18:30:44.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.234+0000 7fcab0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:44.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.234+0000 7fcab0ff9640 10 client.?.objecter ms_handle_connect 0x7fcab40b1730 2024-03-25T18:30:44.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.234+0000 7fcab0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:44.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.240+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061650931 0 0) 0x7fcab4140930 con 0x7fcab40b4d60 2024-03-25T18:30:44.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.240+0000 7fcab0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.240+0000 7fcab0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.240+0000 7fcab0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.240+0000 7fcab0ff9640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:30:44.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.240+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa0004eb0 con 0x7fcab40b4d60 2024-03-25T18:30:44.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.241+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893744429 0 0) 0x7fcab4141870 con 0x7fcab414be90 2024-03-25T18:30:44.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.241+0000 7fcab0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.241+0000 7fcab0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.241+0000 7fcab0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.241+0000 7fcab0ff9640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:30:44.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.241+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab4140930 con 0x7fcab414be90 2024-03-25T18:30:44.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.250+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079962035 0 0) 0x7fcab41327e0 con 0x7fcab40b1730 2024-03-25T18:30:44.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.250+0000 7fcab0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.250+0000 7fcab0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.250+0000 7fcab0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.250+0000 7fcab0ff9640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:30:44.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.250+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab4141870 con 0x7fcab40b1730 2024-03-25T18:30:44.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.268+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2590225591 0 0) 0x7fcaa0004eb0 con 0x7fcab40b4d60 2024-03-25T18:30:44.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.268+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcab41327e0 con 0x7fcab40b4d60 2024-03-25T18:30:44.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.268+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcaa8001990 con 0x7fcab40b4d60 2024-03-25T18:30:44.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.268+0000 7fcab0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.268+0000 7fcab0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:44.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.268+0000 7fcab0ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:44.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.268+0000 7fcab0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.275+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3942220127 0 0) 0x7fcab4140930 con 0x7fcab414be90 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.275+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaa0004eb0 con 0x7fcab414be90 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.275+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcaac001b90 con 0x7fcab414be90 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.275+0000 7fcab0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.275+0000 7fcab0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.275+0000 7fcab0ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.276 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.277 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.277 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:44.277 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:44.277 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:44.277 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:44.277 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:44.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3067111221 0 0) 0x7fcab41327e0 con 0x7fcab40b4d60 2024-03-25T18:30:44.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 >> v1:172.21.15.55:6790/0 conn(0x7fcab40b1730 legacy=0x7fcab41dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 >> v1:172.21.15.133:6789/0 conn(0x7fcab414be90 legacy=0x7fcab41320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:44.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 1 monclient: found mon.a 2024-03-25T18:30:44.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:44.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:44.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab4141ae0 con 0x7fcab40b4d60 2024-03-25T18:30:44.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcab0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.281+0000 7fcaba488640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.282+0000 7fcaba488640 10 monclient: _renew_subs 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.282+0000 7fcaba488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.282+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab4137850 con 0x7fcab40b4d60 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.282+0000 7fcaba488640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.282+0000 7fcaba488640 10 monclient: _renew_subs 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.282+0000 7fcaba488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.282+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab4140560 con 0x7fcab40b4d60 2024-03-25T18:30:44.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.282+0000 7fcaba488640 1 librados: init done 2024-03-25T18:30:44.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.283+0000 7fcaba488640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:44.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.283+0000 7fcaba488640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:44.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.283+0000 7fcaba488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:44.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.283+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab41401f0 con 0x7fcab40b4d60 2024-03-25T18:30:44.611 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.610+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaa8003610 con 0x7fcab40b4d60 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.610+0000 7fcab0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.610+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcaa80039a0 con 0x7fcab40b4d60 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.610+0000 7fcab0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.610+0000 7fcab0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.610+0000 7fcab0ff9640 10 monclient: dump: 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:44.612 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:44.613 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:44.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.610+0000 7fcab27fc640 4 set_mon_vals no callback set 2024-03-25T18:30:44.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.611+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fcaa8003bc0 con 0x7fcab40b4d60 2024-03-25T18:30:44.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.612+0000 7fcab3fff640 10 monclient: get_auth_request con 0x7fcaa006e000 auth_method 0 2024-03-25T18:30:44.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.612+0000 7fcab0ff9640 10 client.4347.objecter ms_handle_connect 0x7fcaa006e000 2024-03-25T18:30:44.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.619+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fcaa80547f0 con 0x7fcab40b4d60 2024-03-25T18:30:44.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.619+0000 7fcab0ff9640 10 client.4347.objecter ms_dispatch 0x7fcab4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:44.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.619+0000 7fcab0ff9640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:44.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.619+0000 7fcab0ff9640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:44.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.619+0000 7fcab0ff9640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T18:30:44.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.619+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcaa8050300 con 0x7fcab40b4d60 2024-03-25T18:30:44.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.619+0000 7fcab0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:44.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.619+0000 7fcab0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:44.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.870+0000 7fcaba488640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:44.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.870+0000 7fcaba488640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:44.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.870+0000 7fcaba488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:44.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.870+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcab40627b0 con 0x7fcab40b4d60 2024-03-25T18:30:44.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.904+0000 7fcab0ff9640 1 -- 172.21.15.55:0/1419819272 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fcaa8052230 con 0x7fcab40b4d60 2024-03-25T18:30:44.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.904+0000 7fcab0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:44.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.904+0000 7fcab0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:44.905 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:30:44.905 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:44.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 10 librados: watch_flush enter 2024-03-25T18:30:44.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 10 librados: watch_flush exit 2024-03-25T18:30:44.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T18:30:44.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 10 client.4347.objecter successfully canceled tick 2024-03-25T18:30:44.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 >> v1:172.21.15.55:6800/286536064 conn(0x7fcaa006e000 legacy=0x7fcaa00704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 10 monclient: shutdown 2024-03-25T18:30:44.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:44.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 >> v1:172.21.15.55:6789/0 conn(0x7fcab40b4d60 legacy=0x7fcab41319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 shutdown_connections 2024-03-25T18:30:44.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 >> 172.21.15.55:0/1419819272 conn(0x7fcab40617a0 msgr2=0x7fcab40b6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:44.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 shutdown_connections 2024-03-25T18:30:44.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 1 -- 172.21.15.55:0/1419819272 wait complete. 2024-03-25T18:30:44.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:44.907+0000 7fcaba488640 1 librados: shutdown 2024-03-25T18:30:44.919 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:51.726 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:51.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.842+0000 7f92905a8640 10 monclient: get_monmap_and_config 2024-03-25T18:30:51.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.842+0000 7f92905a8640 10 monclient: build_initial_monmap 2024-03-25T18:30:51.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.842+0000 7f92905a8640 10 monclient: monmap: 2024-03-25T18:30:51.843 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:51.843 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:51.843 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:30:51.843232+0000 2024-03-25T18:30:51.843 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:30:51.843232+0000 2024-03-25T18:30:51.843 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:51.844 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:51.844 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:51.844 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:30:51.844 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:30:51.844 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:51.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.843+0000 7f92905a8640 1 Processor -- start 2024-03-25T18:30:51.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.843+0000 7f92905a8640 1 -- start start 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.843+0000 7f92905a8640 10 monclient: init 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 10 monclient(hunting): picked mon.noname-b con 0x7f92880a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 10 monclient(hunting): picked mon.noname-c con 0x7f928813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 10 monclient(hunting): picked mon.noname-a con 0x7f92881430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9288142520 con 0x7f92881430c0 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92880b28a0 con 0x7f928813f710 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92880b2a30 con 0x7f92880a98f0 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:51.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.844+0000 7f92905a8640 10 monclient(hunting): authenticate will time out at 7456.102271s 2024-03-25T18:30:51.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.852+0000 7f928db43640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f928813f710 0x7f92881429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:38108/0 (socket says 172.21.15.55:38108) 2024-03-25T18:30:51.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.852+0000 7f928db43640 1 -- 172.21.15.55:0/4043869429 learned_addr learned my addr 172.21.15.55:0/4043869429 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:51.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125400710 0 0) 0x7f92880b28a0 con 0x7f928813f710 2024-03-25T18:30:51.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 10 monclient(hunting): _negotiate 2024-03-25T18:30:51.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:51.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:51.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:30:51.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9274003520 con 0x7f928813f710 2024-03-25T18:30:51.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929277637 0 0) 0x7f9288142520 con 0x7f92881430c0 2024-03-25T18:30:51.870 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 10 monclient(hunting): _negotiate 2024-03-25T18:30:51.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:51.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:51.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:30:51.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.869+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92880b28a0 con 0x7f92881430c0 2024-03-25T18:30:51.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 930471205 0 0) 0x7f9274003520 con 0x7f928813f710 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 >> v1:172.21.15.55:6790/0 conn(0x7f92880a98f0 legacy=0x7f92880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 >> v1:172.21.15.55:6789/0 conn(0x7f92881430c0 legacy=0x7f928814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 10 monclient: _finish_hunting 0 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 1 monclient: found mon.noname-c 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92880b44c0 con 0x7f928813f710 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 10 monclient: _finish_auth 0 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.877+0000 7f928d342640 10 monclient: _check_auth_tickets 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.878+0000 7f928d342640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.878+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9278001690 con 0x7f928813f710 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.878+0000 7f92905a8640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:30:51.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.878+0000 7f92905a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.878+0000 7f928d342640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.878+0000 7f928d342640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.878+0000 7f928d342640 10 monclient: dump: 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:51.880 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:51.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.878+0000 7f92905a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.892+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92780017f0 con 0x7f928813f710 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.892+0000 7f928d342640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.892+0000 7f928d342640 1 -- 172.21.15.55:0/4043869429 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9278001c00 con 0x7f928813f710 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.892+0000 7f92905a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.892+0000 7f92905a8640 4 set_mon_vals no callback set 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.892+0000 7f928d342640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.892+0000 7f928d342640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.892+0000 7f928d342640 10 monclient: dump: 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.893 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.894 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:51.894 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:51.894 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:51.894 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:51.894 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:51.894 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:51.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.893+0000 7f92905a8640 10 monclient: shutdown 2024-03-25T18:30:51.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.893+0000 7f92905a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:51.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.893+0000 7f92905a8640 1 -- 172.21.15.55:0/4043869429 >> v1:172.21.15.133:6789/0 conn(0x7f928813f710 legacy=0x7f92881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.893+0000 7f92905a8640 1 -- 172.21.15.55:0/4043869429 shutdown_connections 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.893+0000 7f92905a8640 1 -- 172.21.15.55:0/4043869429 >> 172.21.15.55:0/4043869429 conn(0x7f92880b7ea0 msgr2=0x7f92880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.893+0000 7f92905a8640 1 -- 172.21.15.55:0/4043869429 shutdown_connections 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.893+0000 7f92905a8640 1 -- 172.21.15.55:0/4043869429 wait complete. 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.894+0000 7f92905a8640 10 monclient: build_initial_monmap 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.894+0000 7f92905a8640 10 monclient: monmap: 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:30:51.896 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:51.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.894+0000 7f92905a8640 1 librados: starting msgr at 2024-03-25T18:30:51.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.894+0000 7f92905a8640 1 librados: starting objecter 2024-03-25T18:30:51.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.894+0000 7f92905a8640 1 Processor -- start 2024-03-25T18:30:51.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.894+0000 7f92905a8640 1 -- start start 2024-03-25T18:30:51.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.894+0000 7f92905a8640 1 librados: setting wanted keys 2024-03-25T18:30:51.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.894+0000 7f92905a8640 1 librados: calling monclient init 2024-03-25T18:30:51.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.894+0000 7f92905a8640 10 monclient: init 2024-03-25T18:30:51.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:51.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:51.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 10 monclient(hunting): picked mon.noname-c con 0x7f92880a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:51.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 10 monclient(hunting): picked mon.noname-a con 0x7f928813f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:51.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 10 monclient(hunting): picked mon.noname-b con 0x7f92881430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:51.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9288138a50 con 0x7f928813f710 2024-03-25T18:30:51.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92880ab9e0 con 0x7f92881430c0 2024-03-25T18:30:51.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92881d7880 con 0x7f92880a98f0 2024-03-25T18:30:51.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:51.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.895+0000 7f92905a8640 10 monclient(hunting): authenticate will time out at 7456.153197s 2024-03-25T18:30:51.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.901+0000 7f928db43640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f928813f710 0x7f92881d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:57972/0 (socket says 172.21.15.55:57972) 2024-03-25T18:30:51.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.901+0000 7f928db43640 1 -- 172.21.15.55:0/1079178177 learned_addr learned my addr 172.21.15.55:0/1079178177 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:51.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.904+0000 7f927effd640 10 client.?.objecter ms_handle_connect 0x7f92880a98f0 2024-03-25T18:30:51.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.904+0000 7f927effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:51.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.911+0000 7f927effd640 10 client.?.objecter ms_handle_connect 0x7f928813f710 2024-03-25T18:30:51.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.911+0000 7f927effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:51.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.914+0000 7f927effd640 10 client.?.objecter ms_handle_connect 0x7f92881430c0 2024-03-25T18:30:51.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.914+0000 7f927effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:51.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.921+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527739114 0 0) 0x7f9288138a50 con 0x7f928813f710 2024-03-25T18:30:51.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.921+0000 7f927effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:51.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.921+0000 7f927effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:51.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.921+0000 7f927effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:51.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.921+0000 7f927effd640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:30:51.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.921+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f926c004eb0 con 0x7f928813f710 2024-03-25T18:30:51.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.923+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308398613 0 0) 0x7f92880ab9e0 con 0x7f92881430c0 2024-03-25T18:30:51.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.923+0000 7f927effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:51.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.923+0000 7f927effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:51.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.923+0000 7f927effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:51.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.923+0000 7f927effd640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:30:51.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.923+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9288138a50 con 0x7f92881430c0 2024-03-25T18:30:51.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.929+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378642181 0 0) 0x7f92881d7880 con 0x7f92880a98f0 2024-03-25T18:30:51.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.929+0000 7f927effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:51.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.929+0000 7f927effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:51.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.929+0000 7f927effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:51.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.929+0000 7f927effd640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:30:51.930 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.929+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92880ab9e0 con 0x7f92880a98f0 2024-03-25T18:30:51.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.937+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 368104017 0 0) 0x7f926c004eb0 con 0x7f928813f710 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.937+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92881d7880 con 0x7f928813f710 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.938+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9278001c00 con 0x7f928813f710 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.938+0000 7f927effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.938+0000 7f927effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.938+0000 7f927effd640 10 monclient(hunting): dump: 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:51.939 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:51.940 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:51.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.938+0000 7f927effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:51.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.944+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 114338878 0 0) 0x7f9288138a50 con 0x7f92881430c0 2024-03-25T18:30:51.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.944+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f926c004eb0 con 0x7f92881430c0 2024-03-25T18:30:51.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.944+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9280001b90 con 0x7f92881430c0 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.944+0000 7f927effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.945+0000 7f927effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.945+0000 7f927effd640 10 monclient(hunting): dump: 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:51.946 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.946+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2999907880 0 0) 0x7f92881d7880 con 0x7f928813f710 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.946+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 >> v1:172.21.15.55:6790/0 conn(0x7f92880a98f0 legacy=0x7f9288138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.946+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 >> v1:172.21.15.133:6789/0 conn(0x7f92881430c0 legacy=0x7f92881e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.946+0000 7f927effd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.946+0000 7f927effd640 1 monclient: found mon.a 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.946+0000 7f927effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.946+0000 7f927effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.946+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92881d84b0 con 0x7f928813f710 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.946+0000 7f927effd640 10 monclient: _finish_auth 0 2024-03-25T18:30:51.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f927effd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f927effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f92905a8640 5 monclient: authenticate success, global_id 4353 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f92905a8640 10 monclient: _renew_subs 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f92905a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92881d51f0 con 0x7f928813f710 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f92905a8640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f92905a8640 10 monclient: _renew_subs 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f92905a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9288138e60 con 0x7f928813f710 2024-03-25T18:30:51.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.947+0000 7f92905a8640 1 librados: init done 2024-03-25T18:30:51.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.948+0000 7f92905a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:51.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.948+0000 7f92905a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:51.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.948+0000 7f92905a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:51.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.948+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92881d76e0 con 0x7f928813f710 2024-03-25T18:30:51.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.969+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9278004a90 con 0x7f928813f710 2024-03-25T18:30:51.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.969+0000 7f927effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.969+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9278004c80 con 0x7f928813f710 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.969+0000 7f927effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.969+0000 7f927effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.969+0000 7f927effd640 10 monclient: dump: 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:51.971 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:51.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.969+0000 7f928cb41640 4 set_mon_vals no callback set 2024-03-25T18:30:51.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.970+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f9278004ea0 con 0x7f928813f710 2024-03-25T18:30:51.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.971+0000 7f928e344640 10 monclient: get_auth_request con 0x7f926c06df70 auth_method 0 2024-03-25T18:30:51.972 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.971+0000 7f927effd640 10 client.4353.objecter ms_handle_connect 0x7f926c06df70 2024-03-25T18:30:51.975 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.975+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f92780547c0 con 0x7f928813f710 2024-03-25T18:30:51.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.975+0000 7f927effd640 10 client.4353.objecter ms_dispatch 0x7f9288000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:51.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.975+0000 7f927effd640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:51.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.975+0000 7f927effd640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:51.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:51.975+0000 7f927effd640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T18:30:52.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.274+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92780502d0 con 0x7f928813f710 2024-03-25T18:30:52.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.274+0000 7f927effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:52.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.274+0000 7f927effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:52.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.501+0000 7f92905a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:52.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.501+0000 7f92905a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:52.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.501+0000 7f92905a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:30:52.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.501+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92881d7a10 con 0x7f928813f710 2024-03-25T18:30:52.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.526+0000 7f927effd640 1 -- 172.21.15.55:0/1079178177 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f9278052200 con 0x7f928813f710 2024-03-25T18:30:52.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.527+0000 7f927effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:52.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.527+0000 7f927effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:52.528 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:30:52.528 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:52.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 10 librados: watch_flush enter 2024-03-25T18:30:52.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 10 librados: watch_flush exit 2024-03-25T18:30:52.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T18:30:52.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 10 client.4353.objecter successfully canceled tick 2024-03-25T18:30:52.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 >> v1:172.21.15.55:6800/286536064 conn(0x7f926c06df70 legacy=0x7f926c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:52.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 10 monclient: shutdown 2024-03-25T18:30:52.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:52.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 >> v1:172.21.15.55:6789/0 conn(0x7f928813f710 legacy=0x7f92881d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:52.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 shutdown_connections 2024-03-25T18:30:52.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.529+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 >> 172.21.15.55:0/1079178177 conn(0x7f92880b7ea0 msgr2=0x7f9288074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:52.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.530+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 shutdown_connections 2024-03-25T18:30:52.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.530+0000 7f92905a8640 1 -- 172.21.15.55:0/1079178177 wait complete. 2024-03-25T18:30:52.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:52.530+0000 7f92905a8640 1 librados: shutdown 2024-03-25T18:30:52.543 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:54.911 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:03:05.081 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:54.919 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:03:05.089 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:54.965 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:05.102 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:54.970 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:03:05.140 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:54.976 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:03:05.146 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:54.981 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:05.119 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:54.988 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:03:05.158 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:55.007 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:05.144 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:55.019 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:03:05.189 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:59.350 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:59.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.460+0000 7fcbe7bba640 10 monclient: get_monmap_and_config 2024-03-25T18:30:59.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.460+0000 7fcbe7bba640 10 monclient: build_initial_monmap 2024-03-25T18:30:59.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.460+0000 7fcbe7bba640 10 monclient: monmap: 2024-03-25T18:30:59.461 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:59.461 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:59.461 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:30:59.461308+0000 2024-03-25T18:30:59.461 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:30:59.461308+0000 2024-03-25T18:30:59.461 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:59.462 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:59.462 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:59.462 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:30:59.462 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:30:59.462 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:59.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.461+0000 7fcbe7bba640 1 Processor -- start 2024-03-25T18:30:59.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.461+0000 7fcbe7bba640 1 -- start start 2024-03-25T18:30:59.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.461+0000 7fcbe7bba640 10 monclient: init 2024-03-25T18:30:59.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.461+0000 7fcbe7bba640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:59.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.461+0000 7fcbe7bba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:59.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.461+0000 7fcbe7bba640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe00b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:59.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.461+0000 7fcbe7bba640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe00b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:59.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.462+0000 7fcbe7bba640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe014be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:59.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.462+0000 7fcbe7bba640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00af940 con 0x7fcbe014be90 2024-03-25T18:30:59.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.462+0000 7fcbe7bba640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00b0a70 con 0x7fcbe00b4d60 2024-03-25T18:30:59.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.462+0000 7fcbe7bba640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00b0d50 con 0x7fcbe00b1730 2024-03-25T18:30:59.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.462+0000 7fcbe7bba640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:59.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.462+0000 7fcbe7bba640 10 monclient(hunting): authenticate will time out at 7463.719497s 2024-03-25T18:30:59.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.479+0000 7fcbe5956640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fcbe00b4d60 0x7fcbe013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:42356/0 (socket says 172.21.15.55:42356) 2024-03-25T18:30:59.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.479+0000 7fcbe5956640 1 -- 172.21.15.55:0/2445382478 learned_addr learned my addr 172.21.15.55:0/2445382478 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:59.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.492+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012106350 0 0) 0x7fcbe00b0a70 con 0x7fcbe00b4d60 2024-03-25T18:30:59.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.492+0000 7fcbe4954640 10 monclient(hunting): _negotiate 2024-03-25T18:30:59.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.492+0000 7fcbe4954640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:59.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.492+0000 7fcbe4954640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:59.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.492+0000 7fcbe4954640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:30:59.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.492+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbcc003520 con 0x7fcbe00b4d60 2024-03-25T18:30:59.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.498+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562924979 0 0) 0x7fcbe00af940 con 0x7fcbe014be90 2024-03-25T18:30:59.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.498+0000 7fcbe4954640 10 monclient(hunting): _negotiate 2024-03-25T18:30:59.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.498+0000 7fcbe4954640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:59.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.498+0000 7fcbe4954640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:59.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.498+0000 7fcbe4954640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:30:59.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.498+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe00b0a70 con 0x7fcbe014be90 2024-03-25T18:30:59.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.503+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42148157 0 0) 0x7fcbe00b0d50 con 0x7fcbe00b1730 2024-03-25T18:30:59.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.503+0000 7fcbe4954640 10 monclient(hunting): _negotiate 2024-03-25T18:30:59.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.503+0000 7fcbe4954640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:59.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.503+0000 7fcbe4954640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:59.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.503+0000 7fcbe4954640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:30:59.505 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.503+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe00af940 con 0x7fcbe00b1730 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 332452660 0 0) 0x7fcbe00af940 con 0x7fcbe00b1730 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 >> v1:172.21.15.133:6789/0 conn(0x7fcbe00b4d60 legacy=0x7fcbe013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 >> v1:172.21.15.55:6789/0 conn(0x7fcbe014be90 legacy=0x7fcbe014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 10 monclient: _finish_hunting 0 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 1 monclient: found mon.noname-b 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe00b1030 con 0x7fcbe00b1730 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 10 monclient: _finish_auth 0 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 10 monclient: _check_auth_tickets 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcbd00015b0 con 0x7fcbe00b1730 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe7bba640 5 monclient: authenticate success, global_id 4352 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.510+0000 7fcbe4954640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:59.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.511+0000 7fcbe4954640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.511+0000 7fcbe4954640 10 monclient: dump: 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:59.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.511+0000 7fcbe7bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.516+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbd0001710 con 0x7fcbe00b1730 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.516+0000 7fcbe4954640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.516+0000 7fcbe4954640 1 -- 172.21.15.55:0/2445382478 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcbd0001b60 con 0x7fcbe00b1730 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.516+0000 7fcbe4954640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.516+0000 7fcbe4954640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.516+0000 7fcbe4954640 10 monclient: dump: 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:59.517 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:59.518 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:59.518 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:59.518 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:59.518 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:59.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.516+0000 7fcbe7bba640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:59.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.516+0000 7fcbe7bba640 4 set_mon_vals no callback set 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.517+0000 7fcbe7bba640 10 monclient: shutdown 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.517+0000 7fcbe7bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.517+0000 7fcbe7bba640 1 -- 172.21.15.55:0/2445382478 >> v1:172.21.15.55:6790/0 conn(0x7fcbe00b1730 legacy=0x7fcbe00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.517+0000 7fcbe7bba640 1 -- 172.21.15.55:0/2445382478 shutdown_connections 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.517+0000 7fcbe7bba640 1 -- 172.21.15.55:0/2445382478 >> 172.21.15.55:0/2445382478 conn(0x7fcbe00617a0 msgr2=0x7fcbe0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.517+0000 7fcbe7bba640 1 -- 172.21.15.55:0/2445382478 shutdown_connections 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.517+0000 7fcbe7bba640 1 -- 172.21.15.55:0/2445382478 wait complete. 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.517+0000 7fcbe7bba640 10 monclient: build_initial_monmap 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.517+0000 7fcbe7bba640 10 monclient: monmap: 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:30:59.520 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.518+0000 7fcbe7bba640 1 librados: starting msgr at 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.518+0000 7fcbe7bba640 1 librados: starting objecter 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.518+0000 7fcbe7bba640 1 Processor -- start 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.518+0000 7fcbe7bba640 1 -- start start 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.518+0000 7fcbe7bba640 1 librados: setting wanted keys 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.518+0000 7fcbe7bba640 1 librados: calling monclient init 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.518+0000 7fcbe7bba640 10 monclient: init 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe00b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:30:59.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe00b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:30:59.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe014be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:30:59.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00aa990 con 0x7fcbe014be90 2024-03-25T18:30:59.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00a9580 con 0x7fcbe00b4d60 2024-03-25T18:30:59.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe0143480 con 0x7fcbe00b1730 2024-03-25T18:30:59.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:59.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.519+0000 7fcbe7bba640 10 monclient(hunting): authenticate will time out at 7463.776509s 2024-03-25T18:30:59.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.524+0000 7fcbe5956640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fcbe00b1730 0x7fcbe01daee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:42700/0 (socket says 172.21.15.55:42700) 2024-03-25T18:30:59.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.524+0000 7fcbe5956640 1 -- 172.21.15.55:0/1335483506 learned_addr learned my addr 172.21.15.55:0/1335483506 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:30:59.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.525+0000 7fcbd67fc640 10 client.?.objecter ms_handle_connect 0x7fcbe00b1730 2024-03-25T18:30:59.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.525+0000 7fcbd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:59.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.526+0000 7fcbd67fc640 10 client.?.objecter ms_handle_connect 0x7fcbe00b4d60 2024-03-25T18:30:59.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.526+0000 7fcbd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:59.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.534+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336813194 0 0) 0x7fcbe00a9580 con 0x7fcbe00b4d60 2024-03-25T18:30:59.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.534+0000 7fcbd67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:59.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.534+0000 7fcbd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:59.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.534+0000 7fcbd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:59.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.534+0000 7fcbd67fc640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:30:59.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.534+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc4004eb0 con 0x7fcbe00b4d60 2024-03-25T18:30:59.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.537+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274164910 0 0) 0x7fcbe0143480 con 0x7fcbe00b1730 2024-03-25T18:30:59.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.537+0000 7fcbd67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:59.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.537+0000 7fcbd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:59.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.537+0000 7fcbd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:59.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.537+0000 7fcbd67fc640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:30:59.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.537+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe00a9580 con 0x7fcbe00b1730 2024-03-25T18:30:59.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.538+0000 7fcbd67fc640 10 client.?.objecter ms_handle_connect 0x7fcbe014be90 2024-03-25T18:30:59.539 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.538+0000 7fcbd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:59.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.551+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703309306 0 0) 0x7fcbe00aa990 con 0x7fcbe014be90 2024-03-25T18:30:59.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.552+0000 7fcbd67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:59.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.552+0000 7fcbd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:59.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.552+0000 7fcbd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:59.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.552+0000 7fcbd67fc640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:30:59.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.552+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe0143480 con 0x7fcbe014be90 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.555+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 270704522 0 0) 0x7fcbe00a9580 con 0x7fcbe00b1730 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.555+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbe00aa990 con 0x7fcbe00b1730 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.555+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcbdc001950 con 0x7fcbe00b1730 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.555+0000 7fcbd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.555+0000 7fcbd67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.555+0000 7fcbd67fc640 10 monclient(hunting): dump: 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:59.556 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:59.557 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:59.557 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:59.557 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:59.557 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:59.557 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:59.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.555+0000 7fcbd67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1756838910 0 0) 0x7fcbe00aa990 con 0x7fcbe00b1730 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 >> v1:172.21.15.133:6789/0 conn(0x7fcbe00b4d60 legacy=0x7fcbe01db5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 >> v1:172.21.15.55:6789/0 conn(0x7fcbe014be90 legacy=0x7fcbe01e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 1 monclient: found mon.c 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe0143610 con 0x7fcbe00b1730 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:59.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:59.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbe7bba640 5 monclient: authenticate success, global_id 4355 2024-03-25T18:30:59.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbe7bba640 10 monclient: _renew_subs 2024-03-25T18:30:59.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbe7bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:59.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.565+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe0142a70 con 0x7fcbe00b1730 2024-03-25T18:30:59.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.566+0000 7fcbe7bba640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:59.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.566+0000 7fcbe7bba640 10 monclient: _renew_subs 2024-03-25T18:30:59.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.566+0000 7fcbe7bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:59.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.566+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe0140560 con 0x7fcbe00b1730 2024-03-25T18:30:59.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.566+0000 7fcbe7bba640 1 librados: init done 2024-03-25T18:30:59.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.567+0000 7fcbe7bba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:59.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.567+0000 7fcbe7bba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:59.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.567+0000 7fcbe7bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:30:59.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.567+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe00a9710 con 0x7fcbe00b1730 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.590+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbdc0047e0 con 0x7fcbe00b1730 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.590+0000 7fcbd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.590+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcbdc004af0 con 0x7fcbe00b1730 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.590+0000 7fcbd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.590+0000 7fcbd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.590+0000 7fcbd67fc640 10 monclient: dump: 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:30:59.591 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:59.592 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:30:59.592 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:30:59.592 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:30:59.592 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:30:59.592 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:30:59.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.590+0000 7fcbd7fff640 4 set_mon_vals no callback set 2024-03-25T18:30:59.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.590+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fcbdc01aac0 con 0x7fcbe00b1730 2024-03-25T18:30:59.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.591+0000 7fcbe5155640 10 monclient: get_auth_request con 0x7fcbc406de60 auth_method 0 2024-03-25T18:30:59.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.591+0000 7fcbd67fc640 10 client.4355.objecter ms_handle_connect 0x7fcbc406de60 2024-03-25T18:30:59.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.595+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fcbdc054830 con 0x7fcbe00b1730 2024-03-25T18:30:59.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.595+0000 7fcbd67fc640 10 client.4355.objecter ms_dispatch 0x7fcbe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:59.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.595+0000 7fcbd67fc640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:59.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.595+0000 7fcbd67fc640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:59.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.595+0000 7fcbd67fc640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-25T18:30:59.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.896+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbdc050170 con 0x7fcbe00b1730 2024-03-25T18:30:59.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.896+0000 7fcbd67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:59.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:30:59.896+0000 7fcbd67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:00.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.155+0000 7fcbe7bba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:00.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.155+0000 7fcbe7bba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:00.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.155+0000 7fcbe7bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:00.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.155+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbe0061d10 con 0x7fcbe00b1730 2024-03-25T18:31:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.348+0000 7fcbd67fc640 1 -- 172.21.15.55:0/1335483506 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fcbdc052270 con 0x7fcbe00b1730 2024-03-25T18:31:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.348+0000 7fcbd67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:00.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.348+0000 7fcbd67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:00.350 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:31:00.350 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:00.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 10 librados: watch_flush enter 2024-03-25T18:31:00.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 10 librados: watch_flush exit 2024-03-25T18:31:00.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-25T18:31:00.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 10 client.4355.objecter successfully canceled tick 2024-03-25T18:31:00.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 >> v1:172.21.15.55:6800/286536064 conn(0x7fcbc406de60 legacy=0x7fcbc4070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:00.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 10 monclient: shutdown 2024-03-25T18:31:00.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:00.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 >> v1:172.21.15.55:6790/0 conn(0x7fcbe00b1730 legacy=0x7fcbe01daee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 shutdown_connections 2024-03-25T18:31:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 >> 172.21.15.55:0/1335483506 conn(0x7fcbe00617a0 msgr2=0x7fcbe013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 shutdown_connections 2024-03-25T18:31:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 1 -- 172.21.15.55:0/1335483506 wait complete. 2024-03-25T18:31:00.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:00.351+0000 7fcbe7bba640 1 librados: shutdown 2024-03-25T18:31:00.364 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:07.171 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:07.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.269+0000 7f10c6057640 10 monclient: get_monmap_and_config 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.269+0000 7f10c6057640 10 monclient: build_initial_monmap 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.269+0000 7f10c6057640 10 monclient: monmap: 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:31:07.271697+0000 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:31:07.271697+0000 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:31:07.272 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:07.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.270+0000 7f10c6057640 1 Processor -- start 2024-03-25T18:31:07.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.270+0000 7f10c6057640 1 -- start start 2024-03-25T18:31:07.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.270+0000 7f10c6057640 10 monclient: init 2024-03-25T18:31:07.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:07.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:07.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 10 monclient(hunting): picked mon.noname-b con 0x7f10c00b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:07.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 10 monclient(hunting): picked mon.noname-a con 0x7f10c00b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:07.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 10 monclient(hunting): picked mon.noname-c con 0x7f10c014be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:07.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00af940 con 0x7f10c00b1730 2024-03-25T18:31:07.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00b0a70 con 0x7f10c014be90 2024-03-25T18:31:07.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00b0d50 con 0x7f10c00b4d60 2024-03-25T18:31:07.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:07.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.271+0000 7f10c6057640 10 monclient(hunting): authenticate will time out at 7471.529630s 2024-03-25T18:31:07.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.276+0000 7f10beffd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f10c00b1730 0x7f10c00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:53980/0 (socket says 172.21.15.55:53980) 2024-03-25T18:31:07.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.276+0000 7f10beffd640 1 -- 172.21.15.55:0/1734636001 learned_addr learned my addr 172.21.15.55:0/1734636001 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:07.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.295+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106363600 0 0) 0x7f10c00b0a70 con 0x7f10c014be90 2024-03-25T18:31:07.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.295+0000 7f10be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:07.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.295+0000 7f10be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:07.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.295+0000 7f10be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:07.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.295+0000 7f10be7fc640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:31:07.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.295+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a8003520 con 0x7f10c014be90 2024-03-25T18:31:07.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.300+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791828323 0 0) 0x7f10c00af940 con 0x7f10c00b1730 2024-03-25T18:31:07.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.300+0000 7f10be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:07.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.300+0000 7f10be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:07.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.300+0000 7f10be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:07.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.300+0000 7f10be7fc640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:31:07.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.300+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c00b0a70 con 0x7f10c00b1730 2024-03-25T18:31:07.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4213523995 0 0) 0x7f10c00b0a70 con 0x7f10c00b1730 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 >> v1:172.21.15.55:6790/0 conn(0x7f10c00b4d60 legacy=0x7f10c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 >> v1:172.21.15.133:6789/0 conn(0x7f10c014be90 legacy=0x7f10c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 1 monclient: found mon.noname-a 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c00b1030 con 0x7f10c00b1730 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:07.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f10b4001670 con 0x7f10c00b1730 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10c6057640 5 monclient: authenticate success, global_id 4362 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10c6057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10be7fc640 10 monclient: dump: 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:07.311 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:07.312 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:07.312 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:07.312 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.307+0000 7f10c6057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.321+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b40017d0 con 0x7f10c00b1730 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.321+0000 7f10be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.321+0000 7f10be7fc640 1 -- 172.21.15.55:0/1734636001 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f10b4001ba0 con 0x7f10c00b1730 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.321+0000 7f10be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.321+0000 7f10be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.321+0000 7f10be7fc640 10 monclient: dump: 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.323 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.324 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:07.324 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:07.324 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:07.324 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:07.324 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:07.324 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:07.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.321+0000 7f10c6057640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:07.324 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.321+0000 7f10c6057640 4 set_mon_vals no callback set 2024-03-25T18:31:07.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.322+0000 7f10c6057640 10 monclient: shutdown 2024-03-25T18:31:07.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.322+0000 7f10c6057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:07.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.322+0000 7f10c6057640 1 -- 172.21.15.55:0/1734636001 >> v1:172.21.15.55:6789/0 conn(0x7f10c00b1730 legacy=0x7f10c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.322+0000 7f10c6057640 1 -- 172.21.15.55:0/1734636001 shutdown_connections 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.322+0000 7f10c6057640 1 -- 172.21.15.55:0/1734636001 >> 172.21.15.55:0/1734636001 conn(0x7f10c00617a0 msgr2=0x7f10c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.322+0000 7f10c6057640 1 -- 172.21.15.55:0/1734636001 shutdown_connections 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.322+0000 7f10c6057640 1 -- 172.21.15.55:0/1734636001 wait complete. 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.323+0000 7f10c6057640 10 monclient: build_initial_monmap 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.323+0000 7f10c6057640 10 monclient: monmap: 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:31:07.326 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.323+0000 7f10c6057640 1 librados: starting msgr at 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.323+0000 7f10c6057640 1 librados: starting objecter 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.323+0000 7f10c6057640 1 Processor -- start 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 1 -- start start 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 1 librados: setting wanted keys 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 1 librados: calling monclient init 2024-03-25T18:31:07.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 10 monclient: init 2024-03-25T18:31:07.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:07.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:07.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 10 monclient(hunting): picked mon.noname-a con 0x7f10c00b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:07.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 10 monclient(hunting): picked mon.noname-c con 0x7f10c00b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:07.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 10 monclient(hunting): picked mon.noname-b con 0x7f10c014be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:07.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c0144b30 con 0x7f10c00b1730 2024-03-25T18:31:07.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c0144cc0 con 0x7f10c014be90 2024-03-25T18:31:07.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c0144e50 con 0x7f10c00b4d60 2024-03-25T18:31:07.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:07.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.324+0000 7f10c6057640 10 monclient(hunting): authenticate will time out at 7471.583253s 2024-03-25T18:31:07.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.335+0000 7f10beffd640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f10c00b4d60 0x7f10c01d78e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:47492/0 (socket says 172.21.15.55:47492) 2024-03-25T18:31:07.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.335+0000 7f10beffd640 1 -- 172.21.15.55:0/1966017713 learned_addr learned my addr 172.21.15.55:0/1966017713 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:07.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.336+0000 7f10a7fff640 10 client.?.objecter ms_handle_connect 0x7f10c00b4d60 2024-03-25T18:31:07.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.336+0000 7f10a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:07.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.340+0000 7f10a7fff640 10 client.?.objecter ms_handle_connect 0x7f10c014be90 2024-03-25T18:31:07.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.340+0000 7f10a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:07.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.348+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372963440 0 0) 0x7f10c0144e50 con 0x7f10c00b4d60 2024-03-25T18:31:07.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.348+0000 7f10a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:07.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.348+0000 7f10a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:07.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.348+0000 7f10a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:07.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.348+0000 7f10a7fff640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:31:07.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.348+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ac004eb0 con 0x7f10c00b4d60 2024-03-25T18:31:07.350 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.348+0000 7f10a7fff640 10 client.?.objecter ms_handle_connect 0x7f10c00b1730 2024-03-25T18:31:07.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.348+0000 7f10a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:07.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.353+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862201438 0 0) 0x7f10c0144b30 con 0x7f10c00b1730 2024-03-25T18:31:07.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.353+0000 7f10a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:07.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.353+0000 7f10a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:07.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.353+0000 7f10a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:07.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.353+0000 7f10a7fff640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:31:07.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.353+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c0144e50 con 0x7f10c00b1730 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667962254 0 0) 0x7f10c0144cc0 con 0x7f10c014be90 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c0144b30 con 0x7f10c014be90 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3602913007 0 0) 0x7f10ac004eb0 con 0x7f10c00b4d60 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10c0144cc0 con 0x7f10c00b4d60 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f10b4001ba0 con 0x7f10c00b4d60 2024-03-25T18:31:07.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 10 monclient(hunting): dump: 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:07.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.358+0000 7f10a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:07.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.372+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3417354405 0 0) 0x7f10c0144e50 con 0x7f10c00b1730 2024-03-25T18:31:07.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.372+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ac004eb0 con 0x7f10c00b1730 2024-03-25T18:31:07.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.372+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f10b00018b0 con 0x7f10c00b1730 2024-03-25T18:31:07.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.372+0000 7f10a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:07.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.372+0000 7f10a7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:07.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.372+0000 7f10a7fff640 10 monclient(hunting): dump: 2024-03-25T18:31:07.374 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:07.375 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:07.375 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.375 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.375 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:07.375 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:07.375 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:07.375 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:07.375 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:07.375 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1029862981 0 0) 0x7f10ac004eb0 con 0x7f10c00b1730 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 >> v1:172.21.15.55:6790/0 conn(0x7f10c00b4d60 legacy=0x7f10c01d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 >> v1:172.21.15.133:6789/0 conn(0x7f10c014be90 legacy=0x7f10c01d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 1 monclient: found mon.a 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c0138e60 con 0x7f10c00b1730 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 10 monclient: _finish_auth 0 2024-03-25T18:31:07.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10c6057640 5 monclient: authenticate success, global_id 4365 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10c6057640 10 monclient: _renew_subs 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10c6057640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c01390b0 con 0x7f10c00b1730 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10c6057640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10c6057640 10 monclient: _renew_subs 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10c6057640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c0137a80 con 0x7f10c00b1730 2024-03-25T18:31:07.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.380+0000 7f10c6057640 1 librados: init done 2024-03-25T18:31:07.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.382+0000 7f10c6057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:07.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.382+0000 7f10c6057640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:07.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.382+0000 7f10c6057640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:31:07.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.382+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c0137dd0 con 0x7f10c00b1730 2024-03-25T18:31:07.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.709+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b00035f0 con 0x7f10c00b1730 2024-03-25T18:31:07.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.709+0000 7f10a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:07.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.709+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f10b0003820 con 0x7f10c00b1730 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.709+0000 7f10a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.709+0000 7f10a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.709+0000 7f10a7fff640 10 monclient: dump: 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:07.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.710+0000 7f10bdffb640 4 set_mon_vals no callback set 2024-03-25T18:31:07.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.710+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f10b0003a40 con 0x7f10c00b1730 2024-03-25T18:31:07.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.711+0000 7f10beffd640 10 monclient: get_auth_request con 0x7f10ac06dfb0 auth_method 0 2024-03-25T18:31:07.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.711+0000 7f10a7fff640 10 client.4365.objecter ms_handle_connect 0x7f10ac06dfb0 2024-03-25T18:31:07.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.717+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f10b0053410 con 0x7f10c00b1730 2024-03-25T18:31:07.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.717+0000 7f10a7fff640 10 client.4365.objecter ms_dispatch 0x7f10c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:07.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.717+0000 7f10a7fff640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:07.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.717+0000 7f10a7fff640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:07.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.717+0000 7f10a7fff640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T18:31:07.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.717+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10b004ef20 con 0x7f10c00b1730 2024-03-25T18:31:07.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.717+0000 7f10a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:07.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.717+0000 7f10a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:07.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.969+0000 7f10c6057640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:07.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.969+0000 7f10c6057640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:07.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.969+0000 7f10c6057640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:31:07.971 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:07.969+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10c00b6fb0 con 0x7f10c00b1730 2024-03-25T18:31:08.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.000+0000 7f10a7fff640 1 -- 172.21.15.55:0/1966017713 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f10b0050e50 con 0x7f10c00b1730 2024-03-25T18:31:08.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.000+0000 7f10a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:08.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.000+0000 7f10a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:08.002 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:31:08.002 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:08.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.002+0000 7f10c6057640 10 librados: watch_flush enter 2024-03-25T18:31:08.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 10 librados: watch_flush exit 2024-03-25T18:31:08.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T18:31:08.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 10 client.4365.objecter successfully canceled tick 2024-03-25T18:31:08.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 >> v1:172.21.15.55:6800/286536064 conn(0x7f10ac06dfb0 legacy=0x7f10ac070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:08.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 10 monclient: shutdown 2024-03-25T18:31:08.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:08.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 >> v1:172.21.15.55:6789/0 conn(0x7f10c00b1730 legacy=0x7f10c01321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:08.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 shutdown_connections 2024-03-25T18:31:08.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 >> 172.21.15.55:0/1966017713 conn(0x7f10c00617a0 msgr2=0x7f10c00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:08.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 shutdown_connections 2024-03-25T18:31:08.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 1 -- 172.21.15.55:0/1966017713 wait complete. 2024-03-25T18:31:08.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:08.003+0000 7f10c6057640 1 librados: shutdown 2024-03-25T18:31:08.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:14.802 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:03:24.972 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.820 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:03:24.991 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.824 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:14.862 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:03:25.032 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.879 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:25.016 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.879 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:03:25.049 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.887 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:03:25.057 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.893 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:25.030 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.920 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:03:25.089 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.924+0000 7f885dad4640 10 monclient: get_monmap_and_config 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.924+0000 7f885dad4640 10 monclient: build_initial_monmap 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.924+0000 7f885dad4640 10 monclient: monmap: 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:31:14.925038+0000 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:31:14.925038+0000 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:31:14.925 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:14.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.925+0000 7f885dad4640 1 Processor -- start 2024-03-25T18:31:14.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.925+0000 7f885dad4640 1 -- start start 2024-03-25T18:31:14.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.925+0000 7f885dad4640 10 monclient: init 2024-03-25T18:31:14.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:14.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:14.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 10 monclient(hunting): picked mon.noname-b con 0x7f88580a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:14.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 10 monclient(hunting): picked mon.noname-c con 0x7f885813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:14.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 10 monclient(hunting): picked mon.noname-a con 0x7f88581430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:14.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8858142520 con 0x7f88581430c0 2024-03-25T18:31:14.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88580b28a0 con 0x7f885813f710 2024-03-25T18:31:14.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88580b2a30 con 0x7f88580a98f0 2024-03-25T18:31:14.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:14.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.926+0000 7f885dad4640 10 monclient(hunting): authenticate will time out at 7479.183527s 2024-03-25T18:31:14.939 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:25.076 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.951+0000 7f8856ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f885813f710 0x7f88581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:51436/0 (socket says 172.21.15.55:51436) 2024-03-25T18:31:14.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.951+0000 7f8856ffd640 1 -- 172.21.15.55:0/1486084636 learned_addr learned my addr 172.21.15.55:0/1486084636 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:14.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.996+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247251551 0 0) 0x7f88580b28a0 con 0x7f885813f710 2024-03-25T18:31:14.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.996+0000 7f88567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:14.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.996+0000 7f88567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:14.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.996+0000 7f88567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:14.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.996+0000 7f88567fc640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:31:14.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.996+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8840003520 con 0x7f885813f710 2024-03-25T18:31:14.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.997+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607100912 0 0) 0x7f8858142520 con 0x7f88581430c0 2024-03-25T18:31:14.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.997+0000 7f88567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:14.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.997+0000 7f88567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:14.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.997+0000 7f88567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:14.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.997+0000 7f88567fc640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:31:14.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:14.997+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88580b28a0 con 0x7f88581430c0 2024-03-25T18:31:15.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4262381490 0 0) 0x7f88580b28a0 con 0x7f88581430c0 2024-03-25T18:31:15.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 >> v1:172.21.15.55:6790/0 conn(0x7f88580a98f0 legacy=0x7f88580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:15.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 >> v1:172.21.15.133:6789/0 conn(0x7f885813f710 legacy=0x7f88581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:15.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 1 monclient: found mon.noname-a 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88580b44c0 con 0x7f88581430c0 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f885dad4640 5 monclient: authenticate success, global_id 4368 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f88500019f0 con 0x7f88581430c0 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f88567fc640 10 monclient: dump: 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.036 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.037 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:15.037 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:15.037 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:15.037 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:15.037 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:15.037 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:15.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.034+0000 7f885dad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.046+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8850001b50 con 0x7f88581430c0 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.046+0000 7f88567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.046+0000 7f88567fc640 1 -- 172.21.15.55:0/1486084636 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8850001f20 con 0x7f88581430c0 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.046+0000 7f88567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.046+0000 7f88567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.046+0000 7f88567fc640 10 monclient: dump: 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.047 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:15.048 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:15.048 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:15.048 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:15.048 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:15.048 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:15.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.046+0000 7f885dad4640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:15.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.046+0000 7f885dad4640 4 set_mon_vals no callback set 2024-03-25T18:31:15.049 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.047+0000 7f885dad4640 10 monclient: shutdown 2024-03-25T18:31:15.049 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.047+0000 7f885dad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:15.049 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.047+0000 7f885dad4640 1 -- 172.21.15.55:0/1486084636 >> v1:172.21.15.55:6789/0 conn(0x7f88581430c0 legacy=0x7f885814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:15.049 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.047+0000 7f885dad4640 1 -- 172.21.15.55:0/1486084636 shutdown_connections 2024-03-25T18:31:15.049 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.047+0000 7f885dad4640 1 -- 172.21.15.55:0/1486084636 >> 172.21.15.55:0/1486084636 conn(0x7f88580b7ea0 msgr2=0x7f88580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:15.049 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.047+0000 7f885dad4640 1 -- 172.21.15.55:0/1486084636 shutdown_connections 2024-03-25T18:31:15.049 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.047+0000 7f885dad4640 1 -- 172.21.15.55:0/1486084636 wait complete. 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.048+0000 7f885dad4640 10 monclient: build_initial_monmap 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.048+0000 7f885dad4640 10 monclient: monmap: 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.048+0000 7f885dad4640 1 librados: starting msgr at 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.048+0000 7f885dad4640 1 librados: starting objecter 2024-03-25T18:31:15.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.049+0000 7f885dad4640 1 Processor -- start 2024-03-25T18:31:15.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.049+0000 7f885dad4640 1 -- start start 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.049+0000 7f885dad4640 1 librados: setting wanted keys 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.049+0000 7f885dad4640 1 librados: calling monclient init 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.049+0000 7f885dad4640 10 monclient: init 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 10 monclient(hunting): picked mon.noname-c con 0x7f88580a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 10 monclient(hunting): picked mon.noname-b con 0x7f885813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 10 monclient(hunting): picked mon.noname-a con 0x7f88581430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88580b78f0 con 0x7f88581430c0 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88580b6e00 con 0x7f885813f710 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88580b6f90 con 0x7f88580a98f0 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:15.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.050+0000 7f885dad4640 10 monclient(hunting): authenticate will time out at 7479.307555s 2024-03-25T18:31:15.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.064+0000 7f8857fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f88581430c0 0x7f88581e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:53646/0 (socket says 172.21.15.55:53646) 2024-03-25T18:31:15.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.064+0000 7f8857fff640 1 -- 172.21.15.55:0/1369698265 learned_addr learned my addr 172.21.15.55:0/1369698265 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:15.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.065+0000 7f883ffff640 10 client.?.objecter ms_handle_connect 0x7f88581430c0 2024-03-25T18:31:15.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.065+0000 7f883ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:15.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.066+0000 7f883ffff640 10 client.?.objecter ms_handle_connect 0x7f885813f710 2024-03-25T18:31:15.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.066+0000 7f883ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:15.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.072+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413703058 0 0) 0x7f88580b6e00 con 0x7f885813f710 2024-03-25T18:31:15.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.072+0000 7f883ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:15.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.072+0000 7f883ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:15.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.072+0000 7f883ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:15.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.072+0000 7f883ffff640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:31:15.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.072+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8844004eb0 con 0x7f885813f710 2024-03-25T18:31:15.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.076+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838980298 0 0) 0x7f88580b78f0 con 0x7f88581430c0 2024-03-25T18:31:15.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.076+0000 7f883ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:15.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.076+0000 7f883ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:15.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.076+0000 7f883ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:15.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.076+0000 7f883ffff640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:31:15.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.076+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88580b6e00 con 0x7f88581430c0 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.086+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4048114997 0 0) 0x7f8844004eb0 con 0x7f885813f710 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.086+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88580b78f0 con 0x7f885813f710 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.086+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f884c001950 con 0x7f885813f710 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.086+0000 7f883ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.086+0000 7f883ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.086+0000 7f883ffff640 10 monclient(hunting): dump: 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:15.087 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:15.088 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:15.088 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:15.088 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:15.088 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:15.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.086+0000 7f883ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:15.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.087+0000 7f883ffff640 10 client.?.objecter ms_handle_connect 0x7f88580a98f0 2024-03-25T18:31:15.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.087+0000 7f883ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:15.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.091+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 821152863 0 0) 0x7f88580b6e00 con 0x7f88581430c0 2024-03-25T18:31:15.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.091+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8844004eb0 con 0x7f88581430c0 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.091+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8850001f60 con 0x7f88581430c0 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.091+0000 7f883ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.091+0000 7f883ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.091+0000 7f883ffff640 10 monclient(hunting): dump: 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:15.093 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 435531365 0 0) 0x7f88580b78f0 con 0x7f885813f710 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 >> v1:172.21.15.55:6790/0 conn(0x7f88580a98f0 legacy=0x7f8858139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 >> v1:172.21.15.55:6789/0 conn(0x7f88581430c0 legacy=0x7f88581e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 1 monclient: found mon.b 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885812f470 con 0x7f885813f710 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 10 monclient: _finish_auth 0 2024-03-25T18:31:15.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f883ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f885dad4640 5 monclient: authenticate success, global_id 4354 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f885dad4640 10 monclient: _renew_subs 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f885dad4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885812f720 con 0x7f885813f710 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f885dad4640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f885dad4640 10 monclient: _renew_subs 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f885dad4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88581db3f0 con 0x7f885813f710 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.093+0000 7f885dad4640 1 librados: init done 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.094+0000 7f885dad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.094+0000 7f885dad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.094+0000 7f885dad4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:15.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.094+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88580b3ef0 con 0x7f885813f710 2024-03-25T18:31:15.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.373+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f884c0047e0 con 0x7f885813f710 2024-03-25T18:31:15.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.373+0000 7f883ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:15.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.373+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f884c004af0 con 0x7f885813f710 2024-03-25T18:31:15.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.373+0000 7f883ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:15.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.373+0000 7f883ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:15.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.373+0000 7f883ffff640 10 monclient: dump: 2024-03-25T18:31:15.375 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:15.375 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.373+0000 7f8855ffb640 4 set_mon_vals no callback set 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.374+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f884c01aac0 con 0x7f885813f710 2024-03-25T18:31:15.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.375+0000 7f88577fe640 10 monclient: get_auth_request con 0x7f884406ddf0 auth_method 0 2024-03-25T18:31:15.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.375+0000 7f883ffff640 10 client.4354.objecter ms_handle_connect 0x7f884406ddf0 2024-03-25T18:31:15.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.379+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f884c054810 con 0x7f885813f710 2024-03-25T18:31:15.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.379+0000 7f883ffff640 10 client.4354.objecter ms_dispatch 0x7f8858000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:15.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.379+0000 7f883ffff640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:15.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.379+0000 7f883ffff640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:15.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.379+0000 7f883ffff640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T18:31:15.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.379+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f884c050150 con 0x7f885813f710 2024-03-25T18:31:15.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.379+0000 7f883ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:15.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.379+0000 7f883ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:15.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.626+0000 7f885dad4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:15.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.626+0000 7f885dad4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:15.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.626+0000 7f885dad4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:15.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.626+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88580b76c0 con 0x7f885813f710 2024-03-25T18:31:15.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.661+0000 7f883ffff640 1 -- 172.21.15.55:0/1369698265 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f88580b76c0 con 0x7f885813f710 2024-03-25T18:31:15.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.661+0000 7f883ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:15.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.661+0000 7f883ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:15.663 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:31:15.663 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 10 librados: watch_flush enter 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 10 librados: watch_flush exit 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 10 client.4354.objecter successfully canceled tick 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 >> v1:172.21.15.55:6800/286536064 conn(0x7f884406ddf0 legacy=0x7f88440702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 10 monclient: shutdown 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 >> v1:172.21.15.133:6789/0 conn(0x7f885813f710 legacy=0x7f88581dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 shutdown_connections 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 >> 172.21.15.55:0/1369698265 conn(0x7f88580b7ea0 msgr2=0x7f8858074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 shutdown_connections 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 1 -- 172.21.15.55:0/1369698265 wait complete. 2024-03-25T18:31:15.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:15.664+0000 7f885dad4640 1 librados: shutdown 2024-03-25T18:31:15.677 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:22.483 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:22.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.599+0000 7f017d8a1640 10 monclient: get_monmap_and_config 2024-03-25T18:31:22.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.599+0000 7f017d8a1640 10 monclient: build_initial_monmap 2024-03-25T18:31:22.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.599+0000 7f017d8a1640 10 monclient: monmap: 2024-03-25T18:31:22.600 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:22.600 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:22.600 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:31:22.600162+0000 2024-03-25T18:31:22.600 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:31:22.600162+0000 2024-03-25T18:31:22.600 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:22.600 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:22.601 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:22.601 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:31:22.601 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:31:22.601 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:22.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 1 Processor -- start 2024-03-25T18:31:22.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 1 -- start start 2024-03-25T18:31:22.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 10 monclient: init 2024-03-25T18:31:22.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:22.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:22.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 10 monclient(hunting): picked mon.noname-c con 0x7f01780a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:22.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 10 monclient(hunting): picked mon.noname-a con 0x7f017813f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:22.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 10 monclient(hunting): picked mon.noname-b con 0x7f01781430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:22.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0178142520 con 0x7f017813f710 2024-03-25T18:31:22.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.600+0000 7f017d8a1640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01780b28a0 con 0x7f01780a98f0 2024-03-25T18:31:22.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.601+0000 7f017d8a1640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01780b2a30 con 0x7f01781430c0 2024-03-25T18:31:22.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.601+0000 7f017d8a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:22.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.601+0000 7f017d8a1640 10 monclient(hunting): authenticate will time out at 7486.858409s 2024-03-25T18:31:22.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.607+0000 7f0176ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f01780a98f0 0x7f01780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:54050/0 (socket says 172.21.15.55:54050) 2024-03-25T18:31:22.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.607+0000 7f0176ffd640 1 -- 172.21.15.55:0/645910758 learned_addr learned my addr 172.21.15.55:0/645910758 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:22.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.617+0000 7f0175ffb640 1 -- 172.21.15.55:0/645910758 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922944387 0 0) 0x7f01780b28a0 con 0x7f01780a98f0 2024-03-25T18:31:22.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.617+0000 7f0175ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:22.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.617+0000 7f0175ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:22.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.617+0000 7f0175ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:22.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.617+0000 7f0175ffb640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:31:22.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.617+0000 7f0175ffb640 1 -- 172.21.15.55:0/645910758 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0160003520 con 0x7f01780a98f0 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 1 -- 172.21.15.55:0/645910758 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3828767827 0 0) 0x7f0160003520 con 0x7f01780a98f0 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 1 -- 172.21.15.55:0/645910758 >> v1:172.21.15.55:6790/0 conn(0x7f01781430c0 legacy=0x7f017814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 1 -- 172.21.15.55:0/645910758 >> v1:172.21.15.55:6789/0 conn(0x7f017813f710 legacy=0x7f01781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 1 monclient: found mon.noname-c 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 1 -- 172.21.15.55:0/645910758 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01780b44c0 con 0x7f01780a98f0 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:22.631 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 1 -- 172.21.15.55:0/645910758 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0168001690 con 0x7f01780a98f0 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f0175ffb640 10 monclient: dump: 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f017d8a1640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:31:22.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.630+0000 7f017d8a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:22.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.639+0000 7f0175ffb640 1 -- 172.21.15.55:0/645910758 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01680017f0 con 0x7f01780a98f0 2024-03-25T18:31:22.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.639+0000 7f0175ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:22.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.639+0000 7f0175ffb640 1 -- 172.21.15.55:0/645910758 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0168001c40 con 0x7f01780a98f0 2024-03-25T18:31:22.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.639+0000 7f0175ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:22.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.639+0000 7f0175ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.639+0000 7f0175ffb640 10 monclient: dump: 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.639+0000 7f017d8a1640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:22.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.639+0000 7f017d8a1640 4 set_mon_vals no callback set 2024-03-25T18:31:22.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.640+0000 7f017d8a1640 10 monclient: shutdown 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.640+0000 7f017d8a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.640+0000 7f017d8a1640 1 -- 172.21.15.55:0/645910758 >> v1:172.21.15.133:6789/0 conn(0x7f01780a98f0 legacy=0x7f01780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.640+0000 7f017d8a1640 1 -- 172.21.15.55:0/645910758 shutdown_connections 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.640+0000 7f017d8a1640 1 -- 172.21.15.55:0/645910758 >> 172.21.15.55:0/645910758 conn(0x7f01780b7ea0 msgr2=0x7f01780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.640+0000 7f017d8a1640 1 -- 172.21.15.55:0/645910758 shutdown_connections 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.640+0000 7f017d8a1640 1 -- 172.21.15.55:0/645910758 wait complete. 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.641+0000 7f017d8a1640 10 monclient: build_initial_monmap 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.641+0000 7f017d8a1640 10 monclient: monmap: 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:22.644 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.641+0000 7f017d8a1640 1 librados: starting msgr at 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.641+0000 7f017d8a1640 1 librados: starting objecter 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.641+0000 7f017d8a1640 1 Processor -- start 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.641+0000 7f017d8a1640 1 -- start start 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.641+0000 7f017d8a1640 1 librados: setting wanted keys 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.641+0000 7f017d8a1640 1 librados: calling monclient init 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.641+0000 7f017d8a1640 10 monclient: init 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:22.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 10 monclient(hunting): picked mon.noname-b con 0x7f01780a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:22.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 10 monclient(hunting): picked mon.noname-c con 0x7f017813f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:22.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 10 monclient(hunting): picked mon.noname-a con 0x7f01781430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:22.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01781db7d0 con 0x7f01781430c0 2024-03-25T18:31:22.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01780b1420 con 0x7f01780a98f0 2024-03-25T18:31:22.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01780b15b0 con 0x7f017813f710 2024-03-25T18:31:22.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:22.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.642+0000 7f017d8a1640 10 monclient(hunting): authenticate will time out at 7486.899691s 2024-03-25T18:31:22.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.654+0000 7f0176ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f01780a98f0 0x7f01781daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:54104/0 (socket says 172.21.15.55:54104) 2024-03-25T18:31:22.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.654+0000 7f0176ffd640 1 -- 172.21.15.55:0/2338450147 learned_addr learned my addr 172.21.15.55:0/2338450147 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:22.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.660+0000 7f015ffff640 10 client.?.objecter ms_handle_connect 0x7f01781430c0 2024-03-25T18:31:22.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.660+0000 7f015ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:22.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.660+0000 7f015ffff640 10 client.?.objecter ms_handle_connect 0x7f017813f710 2024-03-25T18:31:22.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.660+0000 7f015ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:22.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.661+0000 7f015ffff640 10 client.?.objecter ms_handle_connect 0x7f01780a98f0 2024-03-25T18:31:22.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.661+0000 7f015ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:22.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.670+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976241535 0 0) 0x7f01780b15b0 con 0x7f017813f710 2024-03-25T18:31:22.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.670+0000 7f015ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:22.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.670+0000 7f015ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:22.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.670+0000 7f015ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:22.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.670+0000 7f015ffff640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:31:22.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.670+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0164004eb0 con 0x7f017813f710 2024-03-25T18:31:22.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.675+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734321260 0 0) 0x7f01781db7d0 con 0x7f01781430c0 2024-03-25T18:31:22.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.675+0000 7f015ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:22.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.675+0000 7f015ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:22.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.675+0000 7f015ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:22.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.675+0000 7f015ffff640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:31:22.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.675+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01780b15b0 con 0x7f01781430c0 2024-03-25T18:31:22.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.683+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316666903 0 0) 0x7f01780b1420 con 0x7f01780a98f0 2024-03-25T18:31:22.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.683+0000 7f015ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:22.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.683+0000 7f015ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:22.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.683+0000 7f015ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:22.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.683+0000 7f015ffff640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:31:22.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.683+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01781db7d0 con 0x7f01780a98f0 2024-03-25T18:31:22.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.685+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598565343 0 0) 0x7f0164004eb0 con 0x7f017813f710 2024-03-25T18:31:22.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.685+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01780b1420 con 0x7f017813f710 2024-03-25T18:31:22.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.685+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f016c001930 con 0x7f017813f710 2024-03-25T18:31:22.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.685+0000 7f015ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:22.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.685+0000 7f015ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:22.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.685+0000 7f015ffff640 10 monclient(hunting): dump: 2024-03-25T18:31:22.686 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:22.686 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:22.686 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:22.687 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:22.687 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:22.687 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:22.687 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:22.687 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:22.687 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:22.687 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:22.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.685+0000 7f015ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:22.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.689+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1529964791 0 0) 0x7f01780b15b0 con 0x7f01781430c0 2024-03-25T18:31:22.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.689+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0164004eb0 con 0x7f01781430c0 2024-03-25T18:31:22.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.689+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0170001b10 con 0x7f01781430c0 2024-03-25T18:31:22.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.689+0000 7f015ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:22.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.689+0000 7f015ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:22.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.689+0000 7f015ffff640 10 monclient(hunting): dump: 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:22.691 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:22.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 272330543 0 0) 0x7f01780b1420 con 0x7f017813f710 2024-03-25T18:31:22.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 >> v1:172.21.15.133:6789/0 conn(0x7f01780a98f0 legacy=0x7f01781daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 >> v1:172.21.15.55:6789/0 conn(0x7f01781430c0 legacy=0x7f01781d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:22.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:22.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 1 monclient: found mon.c 2024-03-25T18:31:22.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:22.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01780b1740 con 0x7f017813f710 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 10 monclient: _finish_auth 0 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f015ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f017d8a1640 5 monclient: authenticate success, global_id 4370 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f017d8a1640 10 monclient: _renew_subs 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f017d8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01781db1a0 con 0x7f017813f710 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f017d8a1640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f017d8a1640 10 monclient: _renew_subs 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f017d8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01780b08f0 con 0x7f017813f710 2024-03-25T18:31:22.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.694+0000 7f017d8a1640 1 librados: init done 2024-03-25T18:31:22.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.695+0000 7f017d8a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:22.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.695+0000 7f017d8a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:22.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.695+0000 7f017d8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:22.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:22.695+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01780627b0 con 0x7f017813f710 2024-03-25T18:31:23.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.007+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f016c0035f0 con 0x7f017813f710 2024-03-25T18:31:23.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.007+0000 7f015ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:23.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.007+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f016c003980 con 0x7f017813f710 2024-03-25T18:31:23.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.007+0000 7f015ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:23.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.007+0000 7f015ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:23.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.007+0000 7f015ffff640 10 monclient: dump: 2024-03-25T18:31:23.008 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:23.008 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:23.008 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:23.009 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:23.009 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:23.009 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:23.009 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:23.009 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:23.009 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:23.009 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:23.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.007+0000 7f01757fa640 4 set_mon_vals no callback set 2024-03-25T18:31:23.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.008+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f016c003ba0 con 0x7f017813f710 2024-03-25T18:31:23.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.008+0000 7f0176ffd640 10 monclient: get_auth_request con 0x7f016406df70 auth_method 0 2024-03-25T18:31:23.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.008+0000 7f015ffff640 10 client.4370.objecter ms_handle_connect 0x7f016406df70 2024-03-25T18:31:23.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.016+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f016c0535f0 con 0x7f017813f710 2024-03-25T18:31:23.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.016+0000 7f015ffff640 10 client.4370.objecter ms_dispatch 0x7f0178000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:23.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.016+0000 7f015ffff640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:23.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.016+0000 7f015ffff640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:23.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.016+0000 7f015ffff640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-25T18:31:23.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.016+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f016c04f100 con 0x7f017813f710 2024-03-25T18:31:23.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.016+0000 7f015ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:23.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.016+0000 7f015ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:23.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.266+0000 7f017d8a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:23.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.266+0000 7f017d8a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:23.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.266+0000 7f017d8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:23.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.266+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01780abb10 con 0x7f017813f710 2024-03-25T18:31:23.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.418+0000 7f015ffff640 1 -- 172.21.15.55:0/2338450147 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f016c051030 con 0x7f017813f710 2024-03-25T18:31:23.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.418+0000 7f015ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:23.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.418+0000 7f015ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:23.419 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:31:23.419 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 10 librados: watch_flush enter 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 10 librados: watch_flush exit 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 10 client.4370.objecter successfully canceled tick 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 >> v1:172.21.15.55:6800/286536064 conn(0x7f016406df70 legacy=0x7f0164070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 10 monclient: shutdown 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 >> v1:172.21.15.55:6790/0 conn(0x7f017813f710 legacy=0x7f01780b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 shutdown_connections 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.420+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 >> 172.21.15.55:0/2338450147 conn(0x7f01780b7ea0 msgr2=0x7f017813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.421+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 shutdown_connections 2024-03-25T18:31:23.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.421+0000 7f017d8a1640 1 -- 172.21.15.55:0/2338450147 wait complete. 2024-03-25T18:31:23.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:23.421+0000 7f017d8a1640 1 librados: shutdown 2024-03-25T18:31:23.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:30.239 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:30.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.350+0000 7fb1fceb3640 10 monclient: get_monmap_and_config 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.351+0000 7fb1fceb3640 10 monclient: build_initial_monmap 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.351+0000 7fb1fceb3640 10 monclient: monmap: 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:31:30.351889+0000 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:31:30.351889+0000 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:31:30.352 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:30.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.352+0000 7fb1fceb3640 1 Processor -- start 2024-03-25T18:31:30.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.352+0000 7fb1fceb3640 1 -- start start 2024-03-25T18:31:30.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.352+0000 7fb1fceb3640 10 monclient: init 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 10 monclient(hunting): picked mon.noname-a con 0x7fb1f813fc60 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 10 monclient(hunting): picked mon.noname-b con 0x7fb1f8070570 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 10 monclient(hunting): picked mon.noname-c con 0x7fb1f80727a0 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f813c9f0 con 0x7fb1f813fc60 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f806fac0 con 0x7fb1f80727a0 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f80715d0 con 0x7fb1f8070570 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.353+0000 7fb1fceb3640 10 monclient(hunting): authenticate will time out at 7494.610575s 2024-03-25T18:31:30.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.371+0000 7fb1f6d9d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fb1f80727a0 0x7fb1f814a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:59314/0 (socket says 172.21.15.55:59314) 2024-03-25T18:31:30.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.371+0000 7fb1f6d9d640 1 -- 172.21.15.55:0/1301474358 learned_addr learned my addr 172.21.15.55:0/1301474358 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:30.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.384+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126459783 0 0) 0x7fb1f813c9f0 con 0x7fb1f813fc60 2024-03-25T18:31:30.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.384+0000 7fb1f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:30.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.384+0000 7fb1f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:30.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.384+0000 7fb1f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:30.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.384+0000 7fb1f5d9b640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:31:30.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.384+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e4003520 con 0x7fb1f813fc60 2024-03-25T18:31:30.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.388+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736575477 0 0) 0x7fb1f806fac0 con 0x7fb1f80727a0 2024-03-25T18:31:30.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.388+0000 7fb1f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:30.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.388+0000 7fb1f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:30.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.388+0000 7fb1f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:30.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.388+0000 7fb1f5d9b640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:31:30.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.388+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f813c9f0 con 0x7fb1f80727a0 2024-03-25T18:31:30.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2694995803 0 0) 0x7fb1e4003520 con 0x7fb1f813fc60 2024-03-25T18:31:30.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 >> v1:172.21.15.55:6790/0 conn(0x7fb1f8070570 legacy=0x7fb1f8072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:30.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 >> v1:172.21.15.133:6789/0 conn(0x7fb1f80727a0 legacy=0x7fb1f814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:30.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 1 monclient: found mon.noname-a 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f80718b0 con 0x7fb1f813fc60 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb1e8001a10 con 0x7fb1f813fc60 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1fceb3640 5 monclient: authenticate success, global_id 4380 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1fceb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1f5d9b640 10 monclient: dump: 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:30.401 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.402 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.402 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:30.402 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:30.402 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:30.402 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:30.402 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:30.402 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:30.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.399+0000 7fb1fceb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.407+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1e8001b70 con 0x7fb1f813fc60 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.407+0000 7fb1f5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.407+0000 7fb1f5d9b640 1 -- 172.21.15.55:0/1301474358 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb1e8001f80 con 0x7fb1f813fc60 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.407+0000 7fb1f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.407+0000 7fb1f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.407+0000 7fb1f5d9b640 10 monclient: dump: 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.408 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.409 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:30.409 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:30.409 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:30.409 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:30.409 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:30.409 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:30.409 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.407+0000 7fb1fceb3640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:30.409 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.407+0000 7fb1fceb3640 4 set_mon_vals no callback set 2024-03-25T18:31:30.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.408+0000 7fb1fceb3640 10 monclient: shutdown 2024-03-25T18:31:30.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.408+0000 7fb1fceb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:30.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.408+0000 7fb1fceb3640 1 -- 172.21.15.55:0/1301474358 >> v1:172.21.15.55:6789/0 conn(0x7fb1f813fc60 legacy=0x7fb1f806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:30.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.408+0000 7fb1fceb3640 1 -- 172.21.15.55:0/1301474358 shutdown_connections 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.408+0000 7fb1fceb3640 1 -- 172.21.15.55:0/1301474358 >> 172.21.15.55:0/1301474358 conn(0x7fb1f80b4a80 msgr2=0x7fb1f80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.408+0000 7fb1fceb3640 1 -- 172.21.15.55:0/1301474358 shutdown_connections 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.408+0000 7fb1fceb3640 1 -- 172.21.15.55:0/1301474358 wait complete. 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.409+0000 7fb1fceb3640 10 monclient: build_initial_monmap 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.409+0000 7fb1fceb3640 10 monclient: monmap: 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:31:30.411 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.409+0000 7fb1fceb3640 1 librados: starting msgr at 2024-03-25T18:31:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.409+0000 7fb1fceb3640 1 librados: starting objecter 2024-03-25T18:31:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.409+0000 7fb1fceb3640 1 Processor -- start 2024-03-25T18:31:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.409+0000 7fb1fceb3640 1 -- start start 2024-03-25T18:31:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.409+0000 7fb1fceb3640 1 librados: setting wanted keys 2024-03-25T18:31:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.409+0000 7fb1fceb3640 1 librados: calling monclient init 2024-03-25T18:31:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.409+0000 7fb1fceb3640 10 monclient: init 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 10 monclient(hunting): picked mon.noname-c con 0x7fb1f8070570 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 10 monclient(hunting): picked mon.noname-a con 0x7fb1f80727a0 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 10 monclient(hunting): picked mon.noname-b con 0x7fb1f813fc60 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f81409b0 con 0x7fb1f80727a0 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f81c9c60 con 0x7fb1f813fc60 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f81c9f00 con 0x7fb1f8070570 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.410+0000 7fb1fceb3640 10 monclient(hunting): authenticate will time out at 7494.667990s 2024-03-25T18:31:30.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.415+0000 7fb1f659c640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fb1f8070570 0x7fb1f8133060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:57660/0 (socket says 172.21.15.55:57660) 2024-03-25T18:31:30.415 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.415+0000 7fb1f659c640 1 -- 172.21.15.55:0/57315294 learned_addr learned my addr 172.21.15.55:0/57315294 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:30.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.417+0000 7fb1eeffd640 10 client.?.objecter ms_handle_connect 0x7fb1f8070570 2024-03-25T18:31:30.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.417+0000 7fb1eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:30.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.424+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266153326 0 0) 0x7fb1f81c9f00 con 0x7fb1f8070570 2024-03-25T18:31:30.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.424+0000 7fb1eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:30.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.424+0000 7fb1eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:30.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.424+0000 7fb1eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:30.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.424+0000 7fb1eeffd640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:31:30.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.424+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d8004eb0 con 0x7fb1f8070570 2024-03-25T18:31:30.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.429+0000 7fb1eeffd640 10 client.?.objecter ms_handle_connect 0x7fb1f813fc60 2024-03-25T18:31:30.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.430+0000 7fb1eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:30.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.442+0000 7fb1eeffd640 10 client.?.objecter ms_handle_connect 0x7fb1f80727a0 2024-03-25T18:31:30.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.442+0000 7fb1eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:30.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.446+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641265321 0 0) 0x7fb1f81c9c60 con 0x7fb1f813fc60 2024-03-25T18:31:30.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.446+0000 7fb1eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:30.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.446+0000 7fb1eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:30.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.446+0000 7fb1eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:30.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.446+0000 7fb1eeffd640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:31:30.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.446+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f81c9f00 con 0x7fb1f813fc60 2024-03-25T18:31:30.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.459+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4217597740 0 0) 0x7fb1d8004eb0 con 0x7fb1f8070570 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.459+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f81c9c60 con 0x7fb1f8070570 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.459+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb1e8001f80 con 0x7fb1f8070570 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.459+0000 7fb1eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.459+0000 7fb1eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.459+0000 7fb1eeffd640 10 monclient(hunting): dump: 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:30.460 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:30.461 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:30.461 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:30.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.459+0000 7fb1eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:30.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212823084 0 0) 0x7fb1f81409b0 con 0x7fb1f80727a0 2024-03-25T18:31:30.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:30.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:30.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:30.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:31:30.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d8004eb0 con 0x7fb1f80727a0 2024-03-25T18:31:30.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 806332117 0 0) 0x7fb1f81c9f00 con 0x7fb1f813fc60 2024-03-25T18:31:30.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f81409b0 con 0x7fb1f813fc60 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb1f00018f0 con 0x7fb1f813fc60 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.461+0000 7fb1eeffd640 10 monclient(hunting): dump: 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:30.463 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3192899777 0 0) 0x7fb1f81c9c60 con 0x7fb1f8070570 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 >> v1:172.21.15.133:6789/0 conn(0x7fb1f813fc60 legacy=0x7fb1f81e3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 >> v1:172.21.15.55:6789/0 conn(0x7fb1f80727a0 legacy=0x7fb1f81dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 1 monclient: found mon.c 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f81ca090 con 0x7fb1f8070570 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.468+0000 7fb1eeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:30.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1fceb3640 5 monclient: authenticate success, global_id 4376 2024-03-25T18:31:30.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1fceb3640 10 monclient: _renew_subs 2024-03-25T18:31:30.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1fceb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:30.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f81c93c0 con 0x7fb1f8070570 2024-03-25T18:31:30.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1fceb3640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:30.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1fceb3640 10 monclient: _renew_subs 2024-03-25T18:31:30.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1fceb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:30.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f8133950 con 0x7fb1f8070570 2024-03-25T18:31:30.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.469+0000 7fb1fceb3640 1 librados: init done 2024-03-25T18:31:30.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.470+0000 7fb1fceb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:30.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.470+0000 7fb1fceb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:30.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.470+0000 7fb1fceb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:30.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.470+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f80627b0 con 0x7fb1f8070570 2024-03-25T18:31:30.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.684+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1e8001590 con 0x7fb1f8070570 2024-03-25T18:31:30.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.684+0000 7fb1eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:30.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.684+0000 7fb1f559a640 4 set_mon_vals no callback set 2024-03-25T18:31:30.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.785+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb1e8003dc0 con 0x7fb1f8070570 2024-03-25T18:31:30.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.785+0000 7fb1eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.785+0000 7fb1eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.785+0000 7fb1eeffd640 10 monclient: dump: 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:30.786 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:30.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.785+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fb1e8003fa0 con 0x7fb1f8070570 2024-03-25T18:31:30.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.786+0000 7fb1effff640 10 monclient: get_auth_request con 0x7fb1d806dfb0 auth_method 0 2024-03-25T18:31:30.787 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.786+0000 7fb1eeffd640 10 client.4376.objecter ms_handle_connect 0x7fb1d806dfb0 2024-03-25T18:31:30.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.792+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fb1e8054be0 con 0x7fb1f8070570 2024-03-25T18:31:30.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.792+0000 7fb1eeffd640 10 client.4376.objecter ms_dispatch 0x7fb1f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:30.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.792+0000 7fb1eeffd640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:30.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.792+0000 7fb1eeffd640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:30.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.792+0000 7fb1eeffd640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T18:31:30.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.792+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1e8050520 con 0x7fb1f8070570 2024-03-25T18:31:30.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.792+0000 7fb1eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:30.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:30.792+0000 7fb1eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:31.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.035+0000 7fb1fceb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:31.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.035+0000 7fb1fceb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:31.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.035+0000 7fb1fceb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:31.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.035+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1f806b4d0 con 0x7fb1f8070570 2024-03-25T18:31:31.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.123+0000 7fb1eeffd640 1 -- 172.21.15.55:0/57315294 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fb1e8052620 con 0x7fb1f8070570 2024-03-25T18:31:31.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.123+0000 7fb1eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:31.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.123+0000 7fb1eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:31.124 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:31:31.124 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.125+0000 7fb1fceb3640 10 librados: watch_flush enter 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.125+0000 7fb1fceb3640 10 librados: watch_flush exit 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.125+0000 7fb1fceb3640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.125+0000 7fb1fceb3640 10 client.4376.objecter successfully canceled tick 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.126+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 >> v1:172.21.15.55:6800/286536064 conn(0x7fb1d806dfb0 legacy=0x7fb1d8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.126+0000 7fb1fceb3640 10 monclient: shutdown 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.126+0000 7fb1fceb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.126+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 >> v1:172.21.15.55:6790/0 conn(0x7fb1f8070570 legacy=0x7fb1f8133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.126+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 shutdown_connections 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.126+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 >> 172.21.15.55:0/57315294 conn(0x7fb1f80b4a80 msgr2=0x7fb1f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:31.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.126+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 shutdown_connections 2024-03-25T18:31:31.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.126+0000 7fb1fceb3640 1 -- 172.21.15.55:0/57315294 wait complete. 2024-03-25T18:31:31.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:31.126+0000 7fb1fceb3640 1 librados: shutdown 2024-03-25T18:31:31.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:34.871 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:03:45.040 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:34.877 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:03:45.047 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:34.907 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:03:45.078 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:34.928 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:45.065 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:34.934 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:45.071 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:34.951 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:03:45.121 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:34.958 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:03:45.128 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:34.966 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:03:45.104 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:34.983 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:03:45.153 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:37.952 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:38.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.069+0000 7fb278057640 10 monclient: get_monmap_and_config 2024-03-25T18:31:38.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.069+0000 7fb278057640 10 monclient: build_initial_monmap 2024-03-25T18:31:38.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.069+0000 7fb278057640 10 monclient: monmap: 2024-03-25T18:31:38.070 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:38.070 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:38.070 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:31:38.070155+0000 2024-03-25T18:31:38.070 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:31:38.070155+0000 2024-03-25T18:31:38.071 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:38.071 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:38.071 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:38.071 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:31:38.071 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:31:38.071 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:38.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.070+0000 7fb278057640 1 Processor -- start 2024-03-25T18:31:38.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.070+0000 7fb278057640 1 -- start start 2024-03-25T18:31:38.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.070+0000 7fb278057640 10 monclient: init 2024-03-25T18:31:38.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:38.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:38.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 10 monclient(hunting): picked mon.noname-c con 0x7fb2700a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:38.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 10 monclient(hunting): picked mon.noname-b con 0x7fb27013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:38.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 10 monclient(hunting): picked mon.noname-a con 0x7fb2701430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:38.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb270142520 con 0x7fb2701430c0 2024-03-25T18:31:38.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2700b28a0 con 0x7fb2700a98f0 2024-03-25T18:31:38.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2700b2a30 con 0x7fb27013f710 2024-03-25T18:31:38.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:38.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.071+0000 7fb278057640 10 monclient(hunting): authenticate will time out at 7502.328799s 2024-03-25T18:31:38.082 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.082+0000 7fb2765f4640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fb2701430c0 0x7fb27014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:60040/0 (socket says 172.21.15.55:60040) 2024-03-25T18:31:38.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.082+0000 7fb2765f4640 1 -- 172.21.15.55:0/3244890028 learned_addr learned my addr 172.21.15.55:0/3244890028 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:38.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.086+0000 7fb275df3640 1 -- 172.21.15.55:0/3244890028 >> v1:172.21.15.133:6789/0 conn(0x7fb2700a98f0 legacy=0x7fb2700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T18:31:38.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.086+0000 7fb275df3640 1 -- 172.21.15.55:0/3244890028 >> v1:172.21.15.133:6789/0 conn(0x7fb2700a98f0 legacy=0x7fb2700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:31:38.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.086+0000 7fb275df3640 1 --1- 172.21.15.55:0/3244890028 >> v1:172.21.15.133:6789/0 conn(0x7fb2700a98f0 0x7fb2700a9cd0 :-1 s=OPENED pgs=22030 cs=1 l=1).handle_message read tag failed 2024-03-25T18:31:38.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.086+0000 7fb275df3640 1 --1- 172.21.15.55:0/3244890028 >> v1:172.21.15.133:6789/0 conn(0x7fb2700a98f0 0x7fb2700a9cd0 :-1 s=OPENED pgs=22030 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:31:38.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.086+0000 7fb274df1640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.133:6789/0 2024-03-25T18:31:38.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.092+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809841957 0 0) 0x7fb270142520 con 0x7fb2701430c0 2024-03-25T18:31:38.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.092+0000 7fb274df1640 10 monclient(hunting): _negotiate 2024-03-25T18:31:38.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.092+0000 7fb274df1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:38.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.092+0000 7fb274df1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:38.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.092+0000 7fb274df1640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:31:38.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.092+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb25c003520 con 0x7fb2701430c0 2024-03-25T18:31:38.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222932630 0 0) 0x7fb2700b2a30 con 0x7fb27013f710 2024-03-25T18:31:38.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient(hunting): _negotiate 2024-03-25T18:31:38.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:38.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:38.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb270142520 con 0x7fb27013f710 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197656252 0 0) 0x7fb25c003520 con 0x7fb2701430c0 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 >> v1:172.21.15.55:6790/0 conn(0x7fb27013f710 legacy=0x7fb2701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 >> v1:172.21.15.133:6789/0 conn(0x7fb2700a98f0 legacy=0x7fb2700a9cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient: _finish_hunting 0 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 1 monclient: found mon.noname-a 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2700b44c0 con 0x7fb2701430c0 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient: _finish_auth 0 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient: _check_auth_tickets 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:38.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb268001a10 con 0x7fb2701430c0 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb274df1640 10 monclient: dump: 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb278057640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:31:38.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.119+0000 7fb278057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:38.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.147+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb268001b70 con 0x7fb2701430c0 2024-03-25T18:31:38.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.147+0000 7fb274df1640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:38.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.147+0000 7fb274df1640 1 -- 172.21.15.55:0/3244890028 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb268001f80 con 0x7fb2701430c0 2024-03-25T18:31:38.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.147+0000 7fb274df1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:38.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.147+0000 7fb274df1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:38.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.147+0000 7fb274df1640 10 monclient: dump: 2024-03-25T18:31:38.148 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.147+0000 7fb278057640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:38.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.147+0000 7fb278057640 4 set_mon_vals no callback set 2024-03-25T18:31:38.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.148+0000 7fb278057640 10 monclient: shutdown 2024-03-25T18:31:38.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.148+0000 7fb278057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:38.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.148+0000 7fb278057640 1 -- 172.21.15.55:0/3244890028 >> v1:172.21.15.55:6789/0 conn(0x7fb2701430c0 legacy=0x7fb27014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:38.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.148+0000 7fb278057640 1 -- 172.21.15.55:0/3244890028 shutdown_connections 2024-03-25T18:31:38.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.148+0000 7fb278057640 1 -- 172.21.15.55:0/3244890028 >> 172.21.15.55:0/3244890028 conn(0x7fb2700b7ea0 msgr2=0x7fb2700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:38.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.148+0000 7fb278057640 1 -- 172.21.15.55:0/3244890028 shutdown_connections 2024-03-25T18:31:38.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.148+0000 7fb278057640 1 -- 172.21.15.55:0/3244890028 wait complete. 2024-03-25T18:31:38.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.149+0000 7fb278057640 10 monclient: build_initial_monmap 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.149+0000 7fb278057640 10 monclient: monmap: 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.149+0000 7fb278057640 1 librados: starting msgr at 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.149+0000 7fb278057640 1 librados: starting objecter 2024-03-25T18:31:38.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.149+0000 7fb278057640 1 Processor -- start 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.149+0000 7fb278057640 1 -- start start 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.149+0000 7fb278057640 1 librados: setting wanted keys 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.149+0000 7fb278057640 1 librados: calling monclient init 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.149+0000 7fb278057640 10 monclient: init 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 10 monclient(hunting): picked mon.noname-b con 0x7fb2700a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 10 monclient(hunting): picked mon.noname-c con 0x7fb27013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 10 monclient(hunting): picked mon.noname-a con 0x7fb2701430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2701369b0 con 0x7fb2701430c0 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2701366b0 con 0x7fb2700a98f0 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb270133ad0 con 0x7fb27013f710 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:38.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.150+0000 7fb278057640 10 monclient(hunting): authenticate will time out at 7502.407920s 2024-03-25T18:31:38.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.156+0000 7fb275df3640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fb2700a98f0 0x7fb2701371e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:35604/0 (socket says 172.21.15.55:35604) 2024-03-25T18:31:38.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.156+0000 7fb275df3640 1 -- 172.21.15.55:0/3780238143 learned_addr learned my addr 172.21.15.55:0/3780238143 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:38.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.157+0000 7fb2667fc640 10 client.?.objecter ms_handle_connect 0x7fb2700a98f0 2024-03-25T18:31:38.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.157+0000 7fb2667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.172+0000 7fb2667fc640 10 client.?.objecter ms_handle_connect 0x7fb27013f710 2024-03-25T18:31:38.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.172+0000 7fb2667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.175+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368054167 0 0) 0x7fb2701366b0 con 0x7fb2700a98f0 2024-03-25T18:31:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.176+0000 7fb2667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:38.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.176+0000 7fb2667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.176+0000 7fb2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.176+0000 7fb2667fc640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:31:38.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.176+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb254004eb0 con 0x7fb2700a98f0 2024-03-25T18:31:38.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.180+0000 7fb2667fc640 10 client.?.objecter ms_handle_connect 0x7fb2701430c0 2024-03-25T18:31:38.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.180+0000 7fb2667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.189+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713861889 0 0) 0x7fb270133ad0 con 0x7fb27013f710 2024-03-25T18:31:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.189+0000 7fb2667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.189+0000 7fb2667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.189+0000 7fb2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.189+0000 7fb2667fc640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:31:38.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.189+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2701366b0 con 0x7fb27013f710 2024-03-25T18:31:38.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419906272 0 0) 0x7fb2701369b0 con 0x7fb2701430c0 2024-03-25T18:31:38.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:38.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:38.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb270133ad0 con 0x7fb2701430c0 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3261528364 0 0) 0x7fb254004eb0 con 0x7fb2700a98f0 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2701369b0 con 0x7fb2700a98f0 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb26c001a20 con 0x7fb2700a98f0 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 10 monclient(hunting): dump: 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:38.200 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:38.201 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:38.201 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:38.201 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:38.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.198+0000 7fb2667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:38.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1108145289 0 0) 0x7fb2701369b0 con 0x7fb2700a98f0 2024-03-25T18:31:38.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 >> v1:172.21.15.55:6790/0 conn(0x7fb27013f710 legacy=0x7fb270132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:38.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 >> v1:172.21.15.55:6789/0 conn(0x7fb2701430c0 legacy=0x7fb2701333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:38.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:38.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 1 monclient: found mon.b 2024-03-25T18:31:38.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb270136e40 con 0x7fb2700a98f0 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb2667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb278057640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb278057640 10 monclient: _renew_subs 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb278057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2701378f0 con 0x7fb2700a98f0 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb278057640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb278057640 10 monclient: _renew_subs 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb278057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2700b1480 con 0x7fb2700a98f0 2024-03-25T18:31:38.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.217+0000 7fb278057640 1 librados: init done 2024-03-25T18:31:38.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.218+0000 7fb278057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:38.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.218+0000 7fb278057640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:38.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.218+0000 7fb278057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:38.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.218+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb270137030 con 0x7fb2700a98f0 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.492+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb26c001ce0 con 0x7fb2700a98f0 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.492+0000 7fb2667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.492+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb26c003a40 con 0x7fb2700a98f0 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.492+0000 7fb2667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.492+0000 7fb2667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.492+0000 7fb267fff640 4 set_mon_vals no callback set 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.492+0000 7fb2667fc640 10 monclient: dump: 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:38.493 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:38.494 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:38.494 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:38.494 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:38.494 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:38.494 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:38.494 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:38.494 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:38.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.493+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fb26c003c60 con 0x7fb2700a98f0 2024-03-25T18:31:38.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.493+0000 7fb2755f2640 10 monclient: get_auth_request con 0x7fb25406de00 auth_method 0 2024-03-25T18:31:38.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.494+0000 7fb2667fc640 10 client.4372.objecter ms_handle_connect 0x7fb25406de00 2024-03-25T18:31:38.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.496+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fb26c0548f0 con 0x7fb2700a98f0 2024-03-25T18:31:38.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.496+0000 7fb2667fc640 10 client.4372.objecter ms_dispatch 0x7fb270000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:38.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.496+0000 7fb2667fc640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:38.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.496+0000 7fb2667fc640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:38.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.496+0000 7fb2667fc640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T18:31:38.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.497+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb26c050400 con 0x7fb2700a98f0 2024-03-25T18:31:38.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.497+0000 7fb2667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:38.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.497+0000 7fb2667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:38.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.719+0000 7fb278057640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:38.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.719+0000 7fb278057640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:38.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.719+0000 7fb278057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:38.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.719+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb270137a80 con 0x7fb2700a98f0 2024-03-25T18:31:38.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.759+0000 7fb2667fc640 1 -- 172.21.15.55:0/3780238143 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fb26c052330 con 0x7fb2700a98f0 2024-03-25T18:31:38.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.759+0000 7fb2667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:38.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.759+0000 7fb2667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:38.761 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:31:38.761 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:38.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 10 librados: watch_flush enter 2024-03-25T18:31:38.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 10 librados: watch_flush exit 2024-03-25T18:31:38.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 10 client.4372.objecter successfully canceled tick 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 >> v1:172.21.15.55:6800/286536064 conn(0x7fb25406de00 legacy=0x7fb2540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 10 monclient: shutdown 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 >> v1:172.21.15.133:6789/0 conn(0x7fb2700a98f0 legacy=0x7fb2701371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 shutdown_connections 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 >> 172.21.15.55:0/3780238143 conn(0x7fb2700b7ea0 msgr2=0x7fb270074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 shutdown_connections 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 1 -- 172.21.15.55:0/3780238143 wait complete. 2024-03-25T18:31:38.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:38.762+0000 7fb278057640 1 librados: shutdown 2024-03-25T18:31:38.776 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:45.583 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.693+0000 7fbd778d6640 10 monclient: get_monmap_and_config 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.693+0000 7fbd778d6640 10 monclient: build_initial_monmap 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.693+0000 7fbd778d6640 10 monclient: monmap: 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:31:45.694754+0000 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:31:45.694754+0000 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:45.695 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:31:45.696 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:31:45.696 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:45.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.694+0000 7fbd778d6640 1 Processor -- start 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.694+0000 7fbd778d6640 1 -- start start 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.694+0000 7fbd778d6640 10 monclient: init 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 10 monclient(hunting): picked mon.noname-b con 0x7fbd700a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 10 monclient(hunting): picked mon.noname-a con 0x7fbd7013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 10 monclient(hunting): picked mon.noname-c con 0x7fbd701430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd70142520 con 0x7fbd7013f710 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd700b28a0 con 0x7fbd701430c0 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd700b2a30 con 0x7fbd700a98f0 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:45.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.695+0000 7fbd778d6640 10 monclient(hunting): authenticate will time out at 7509.953093s 2024-03-25T18:31:45.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.703+0000 7fbd75e73640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fbd701430c0 0x7fbd7014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:37894/0 (socket says 172.21.15.55:37894) 2024-03-25T18:31:45.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.703+0000 7fbd75e73640 1 -- 172.21.15.55:0/1218928899 learned_addr learned my addr 172.21.15.55:0/1218928899 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:45.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.720+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411635150 0 0) 0x7fbd700b2a30 con 0x7fbd700a98f0 2024-03-25T18:31:45.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.720+0000 7fbd67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.720+0000 7fbd67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.720+0000 7fbd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.720+0000 7fbd67fff640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:31:45.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.720+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd5c003520 con 0x7fbd700a98f0 2024-03-25T18:31:45.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.721+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986152405 0 0) 0x7fbd700b28a0 con 0x7fbd701430c0 2024-03-25T18:31:45.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.721+0000 7fbd67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.721+0000 7fbd67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.721+0000 7fbd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.721+0000 7fbd67fff640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:31:45.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.721+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd700b2a30 con 0x7fbd701430c0 2024-03-25T18:31:45.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666675175 0 0) 0x7fbd5c003520 con 0x7fbd700a98f0 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 >> v1:172.21.15.133:6789/0 conn(0x7fbd701430c0 legacy=0x7fbd7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 >> v1:172.21.15.55:6789/0 conn(0x7fbd7013f710 legacy=0x7fbd701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 1 monclient: found mon.noname-b 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd700b44c0 con 0x7fbd700a98f0 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 10 monclient: _finish_auth 0 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbd6c001690 con 0x7fbd700a98f0 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd778d6640 5 monclient: authenticate success, global_id 4385 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd778d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:45.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd67fff640 10 monclient: dump: 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:45.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.730+0000 7fbd778d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:45.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.740+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd6c0017f0 con 0x7fbd700a98f0 2024-03-25T18:31:45.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.740+0000 7fbd67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:45.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.740+0000 7fbd67fff640 1 -- 172.21.15.55:0/1218928899 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbd6c001c00 con 0x7fbd700a98f0 2024-03-25T18:31:45.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.740+0000 7fbd778d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:45.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.740+0000 7fbd778d6640 4 set_mon_vals no callback set 2024-03-25T18:31:45.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.740+0000 7fbd67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:45.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.740+0000 7fbd67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:45.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.740+0000 7fbd67fff640 10 monclient: dump: 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:45.742 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:45.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.741+0000 7fbd778d6640 10 monclient: shutdown 2024-03-25T18:31:45.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.741+0000 7fbd778d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:45.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.741+0000 7fbd778d6640 1 -- 172.21.15.55:0/1218928899 >> v1:172.21.15.55:6790/0 conn(0x7fbd700a98f0 legacy=0x7fbd700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.741+0000 7fbd778d6640 1 -- 172.21.15.55:0/1218928899 shutdown_connections 2024-03-25T18:31:45.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.741+0000 7fbd778d6640 1 -- 172.21.15.55:0/1218928899 >> 172.21.15.55:0/1218928899 conn(0x7fbd700b7ea0 msgr2=0x7fbd700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:45.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.741+0000 7fbd778d6640 1 -- 172.21.15.55:0/1218928899 shutdown_connections 2024-03-25T18:31:45.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.741+0000 7fbd778d6640 1 -- 172.21.15.55:0/1218928899 wait complete. 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.742+0000 7fbd778d6640 10 monclient: build_initial_monmap 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.742+0000 7fbd778d6640 10 monclient: monmap: 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:31:45.744 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.743+0000 7fbd778d6640 1 librados: starting msgr at 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.743+0000 7fbd778d6640 1 librados: starting objecter 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.743+0000 7fbd778d6640 1 Processor -- start 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.743+0000 7fbd778d6640 1 -- start start 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.743+0000 7fbd778d6640 1 librados: setting wanted keys 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.743+0000 7fbd778d6640 1 librados: calling monclient init 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.743+0000 7fbd778d6640 10 monclient: init 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.743+0000 7fbd778d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.744+0000 7fbd778d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.744+0000 7fbd778d6640 10 monclient(hunting): picked mon.noname-b con 0x7fbd7013f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:45.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.744+0000 7fbd778d6640 10 monclient(hunting): picked mon.noname-a con 0x7fbd701430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:45.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.744+0000 7fbd778d6640 10 monclient(hunting): picked mon.noname-c con 0x7fbd7013bfc0 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:45.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.744+0000 7fbd778d6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd70137320 con 0x7fbd701430c0 2024-03-25T18:31:45.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.744+0000 7fbd778d6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7013add0 con 0x7fbd7013f710 2024-03-25T18:31:45.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.744+0000 7fbd778d6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd70137ec0 con 0x7fbd7013bfc0 2024-03-25T18:31:45.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.744+0000 7fbd778d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:45.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.744+0000 7fbd778d6640 10 monclient(hunting): authenticate will time out at 7510.001983s 2024-03-25T18:31:45.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.756+0000 7fbd74e71640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fbd701430c0 0x7fbd701377b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:58296/0 (socket says 172.21.15.55:58296) 2024-03-25T18:31:45.757 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.756+0000 7fbd74e71640 1 -- 172.21.15.55:0/225630251 learned_addr learned my addr 172.21.15.55:0/225630251 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:45.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.757+0000 7fbd65ffb640 10 client.?.objecter ms_handle_connect 0x7fbd701430c0 2024-03-25T18:31:45.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.757+0000 7fbd65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:45.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.758+0000 7fbd65ffb640 10 client.?.objecter ms_handle_connect 0x7fbd7013f710 2024-03-25T18:31:45.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.758+0000 7fbd65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:45.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.760+0000 7fbd65ffb640 10 client.?.objecter ms_handle_connect 0x7fbd7013bfc0 2024-03-25T18:31:45.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.760+0000 7fbd65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:45.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.768+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629835420 0 0) 0x7fbd70137ec0 con 0x7fbd7013bfc0 2024-03-25T18:31:45.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.768+0000 7fbd65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.768+0000 7fbd65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.768+0000 7fbd65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.768+0000 7fbd65ffb640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:31:45.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.768+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd54004eb0 con 0x7fbd7013bfc0 2024-03-25T18:31:45.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.770+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750456303 0 0) 0x7fbd7013add0 con 0x7fbd7013f710 2024-03-25T18:31:45.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.770+0000 7fbd65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.770+0000 7fbd65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.770+0000 7fbd65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.770+0000 7fbd65ffb640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:31:45.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.770+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd70137ec0 con 0x7fbd7013f710 2024-03-25T18:31:45.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.771+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021512951 0 0) 0x7fbd70137320 con 0x7fbd701430c0 2024-03-25T18:31:45.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.771+0000 7fbd65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:45.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.771+0000 7fbd65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:45.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.771+0000 7fbd65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:45.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.771+0000 7fbd65ffb640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:31:45.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.771+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd7013add0 con 0x7fbd701430c0 2024-03-25T18:31:45.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.779+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1608113956 0 0) 0x7fbd54004eb0 con 0x7fbd7013bfc0 2024-03-25T18:31:45.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.779+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd70137320 con 0x7fbd7013bfc0 2024-03-25T18:31:45.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.779+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbd68001bd0 con 0x7fbd7013bfc0 2024-03-25T18:31:45.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.779+0000 7fbd65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:45.780 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.779+0000 7fbd65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.779+0000 7fbd65ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:45.781 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.779+0000 7fbd65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.783+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 917525771 0 0) 0x7fbd7013add0 con 0x7fbd701430c0 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.783+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd54004eb0 con 0x7fbd701430c0 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.783+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbd60001970 con 0x7fbd701430c0 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.783+0000 7fbd65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.783+0000 7fbd65ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.783+0000 7fbd65ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:45.784 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:45.785 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:45.785 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:45.785 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:45.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 823889502 0 0) 0x7fbd70137320 con 0x7fbd7013bfc0 2024-03-25T18:31:45.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 >> v1:172.21.15.133:6789/0 conn(0x7fbd7013f710 legacy=0x7fbd70135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 >> v1:172.21.15.55:6789/0 conn(0x7fbd701430c0 legacy=0x7fbd701377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:45.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:45.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 1 monclient: found mon.c 2024-03-25T18:31:45.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:45.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd701360e0 con 0x7fbd7013bfc0 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd778d6640 5 monclient: authenticate success, global_id 4388 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd778d6640 10 monclient: _renew_subs 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd778d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7013b2f0 con 0x7fbd7013bfc0 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd778d6640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd778d6640 10 monclient: _renew_subs 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd778d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd701e3bc0 con 0x7fbd7013bfc0 2024-03-25T18:31:45.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.787+0000 7fbd778d6640 1 librados: init done 2024-03-25T18:31:45.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.788+0000 7fbd778d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:45.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.788+0000 7fbd778d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:45.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.788+0000 7fbd778d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:45.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:45.788+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd700b3ef0 con 0x7fbd7013bfc0 2024-03-25T18:31:46.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.119+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd68002080 con 0x7fbd7013bfc0 2024-03-25T18:31:46.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.119+0000 7fbd65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:46.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.119+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbd68004d00 con 0x7fbd7013bfc0 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.119+0000 7fbd65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.119+0000 7fbd65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.119+0000 7fbd65ffb640 10 monclient: dump: 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:46.121 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:46.122 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:46.122 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:46.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.119+0000 7fbd677fe640 4 set_mon_vals no callback set 2024-03-25T18:31:46.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.120+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fbd6801ad90 con 0x7fbd7013bfc0 2024-03-25T18:31:46.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.121+0000 7fbd75672640 10 monclient: get_auth_request con 0x7fbd5406df70 auth_method 0 2024-03-25T18:31:46.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.121+0000 7fbd65ffb640 10 client.4388.objecter ms_handle_connect 0x7fbd5406df70 2024-03-25T18:31:46.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.126+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fbd680549e0 con 0x7fbd7013bfc0 2024-03-25T18:31:46.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.126+0000 7fbd65ffb640 10 client.4388.objecter ms_dispatch 0x7fbd70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:46.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.126+0000 7fbd65ffb640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:46.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.126+0000 7fbd65ffb640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:46.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.126+0000 7fbd65ffb640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T18:31:46.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.126+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd68050320 con 0x7fbd7013bfc0 2024-03-25T18:31:46.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.126+0000 7fbd65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:46.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.126+0000 7fbd65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:46.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.355+0000 7fbd778d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:46.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.355+0000 7fbd778d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:46.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.355+0000 7fbd778d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:31:46.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.355+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd700627b0 con 0x7fbd7013bfc0 2024-03-25T18:31:46.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.389+0000 7fbd65ffb640 1 -- 172.21.15.55:0/225630251 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fbd68052420 con 0x7fbd7013bfc0 2024-03-25T18:31:46.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.389+0000 7fbd65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:46.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.389+0000 7fbd65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:46.390 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:31:46.390 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:46.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 10 librados: watch_flush enter 2024-03-25T18:31:46.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 10 librados: watch_flush exit 2024-03-25T18:31:46.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T18:31:46.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 10 client.4388.objecter successfully canceled tick 2024-03-25T18:31:46.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 >> v1:172.21.15.55:6800/286536064 conn(0x7fbd5406df70 legacy=0x7fbd54070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:46.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 10 monclient: shutdown 2024-03-25T18:31:46.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:46.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 >> v1:172.21.15.55:6790/0 conn(0x7fbd7013bfc0 legacy=0x7fbd701e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:46.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 shutdown_connections 2024-03-25T18:31:46.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.391+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 >> 172.21.15.55:0/225630251 conn(0x7fbd700b7ea0 msgr2=0x7fbd70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:46.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.392+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 shutdown_connections 2024-03-25T18:31:46.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.392+0000 7fbd778d6640 1 -- 172.21.15.55:0/225630251 wait complete. 2024-03-25T18:31:46.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:46.392+0000 7fbd778d6640 1 librados: shutdown 2024-03-25T18:31:46.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:53.213 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:53.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.330+0000 7fb3f190e640 10 monclient: get_monmap_and_config 2024-03-25T18:31:53.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.330+0000 7fb3f190e640 10 monclient: build_initial_monmap 2024-03-25T18:31:53.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.330+0000 7fb3f190e640 10 monclient: monmap: 2024-03-25T18:31:53.332 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:53.332 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:53.332 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:31:53.332278+0000 2024-03-25T18:31:53.333 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:31:53.332278+0000 2024-03-25T18:31:53.333 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:53.333 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:53.333 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:53.333 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:31:53.333 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:31:53.333 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.331+0000 7fb3f190e640 1 Processor -- start 2024-03-25T18:31:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.331+0000 7fb3f190e640 1 -- start start 2024-03-25T18:31:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.331+0000 7fb3f190e640 10 monclient: init 2024-03-25T18:31:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:53.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:53.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3ec0a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:53.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 10 monclient(hunting): picked mon.noname-a con 0x7fb3ec13f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:53.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3ec1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:53.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec142520 con 0x7fb3ec13f710 2024-03-25T18:31:53.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec0b28a0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec0b2a30 con 0x7fb3ec0a98f0 2024-03-25T18:31:53.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:53.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.332+0000 7fb3f190e640 10 monclient(hunting): authenticate will time out at 7517.590716s 2024-03-25T18:31:53.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.338+0000 7fb3eaffd640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fb3ec0a98f0 0x7fb3ec0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:46850/0 (socket says 172.21.15.55:46850) 2024-03-25T18:31:53.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.338+0000 7fb3eaffd640 1 -- 172.21.15.55:0/2884457699 learned_addr learned my addr 172.21.15.55:0/2884457699 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:53.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.352+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654603969 0 0) 0x7fb3ec0b2a30 con 0x7fb3ec0a98f0 2024-03-25T18:31:53.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.352+0000 7fb3ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:53.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.352+0000 7fb3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:53.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.352+0000 7fb3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:53.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.352+0000 7fb3ea7fc640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:31:53.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.352+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d8003520 con 0x7fb3ec0a98f0 2024-03-25T18:31:53.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.365+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823689568 0 0) 0x7fb3ec0b28a0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.365+0000 7fb3ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:53.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.365+0000 7fb3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:53.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.365+0000 7fb3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:53.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.365+0000 7fb3ea7fc640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:31:53.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.365+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ec0b2a30 con 0x7fb3ec1430c0 2024-03-25T18:31:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.369+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829242029 0 0) 0x7fb3ec142520 con 0x7fb3ec13f710 2024-03-25T18:31:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.369+0000 7fb3ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.369+0000 7fb3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.369+0000 7fb3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.369+0000 7fb3ea7fc640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:31:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.369+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ec0b28a0 con 0x7fb3ec13f710 2024-03-25T18:31:53.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.386+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1323007797 0 0) 0x7fb3ec0b2a30 con 0x7fb3ec1430c0 2024-03-25T18:31:53.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.386+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 >> v1:172.21.15.55:6790/0 conn(0x7fb3ec0a98f0 legacy=0x7fb3ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:53.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.386+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 >> v1:172.21.15.55:6789/0 conn(0x7fb3ec13f710 legacy=0x7fb3ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:53.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.386+0000 7fb3ea7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:53.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.386+0000 7fb3ea7fc640 1 monclient: found mon.noname-c 2024-03-25T18:31:53.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.386+0000 7fb3ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:53.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.386+0000 7fb3ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:31:53.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.386+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec0b44c0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.386+0000 7fb3ea7fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.387+0000 7fb3ea7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.387+0000 7fb3ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.387+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb3e40019f0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.387+0000 7fb3f190e640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.387+0000 7fb3f190e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.387+0000 7fb3ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.387+0000 7fb3ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.387+0000 7fb3ea7fc640 10 monclient: dump: 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.390 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.391 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:53.391 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:53.391 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:53.391 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:53.391 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:53.391 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:53.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.387+0000 7fb3f190e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:53.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.394+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3e4001c10 con 0x7fb3ec1430c0 2024-03-25T18:31:53.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.394+0000 7fb3ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:53.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.394+0000 7fb3ea7fc640 1 -- 172.21.15.55:0/2884457699 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb3e4001fa0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.394+0000 7fb3f190e640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:53.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.394+0000 7fb3f190e640 4 set_mon_vals no callback set 2024-03-25T18:31:53.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.394+0000 7fb3ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.394+0000 7fb3ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.394+0000 7fb3ea7fc640 10 monclient: dump: 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:53.397 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.395+0000 7fb3f190e640 10 monclient: shutdown 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.395+0000 7fb3f190e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.395+0000 7fb3f190e640 1 -- 172.21.15.55:0/2884457699 >> v1:172.21.15.133:6789/0 conn(0x7fb3ec1430c0 legacy=0x7fb3ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.395+0000 7fb3f190e640 1 -- 172.21.15.55:0/2884457699 shutdown_connections 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.395+0000 7fb3f190e640 1 -- 172.21.15.55:0/2884457699 >> 172.21.15.55:0/2884457699 conn(0x7fb3ec0b7ea0 msgr2=0x7fb3ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.395+0000 7fb3f190e640 1 -- 172.21.15.55:0/2884457699 shutdown_connections 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.395+0000 7fb3f190e640 1 -- 172.21.15.55:0/2884457699 wait complete. 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.396+0000 7fb3f190e640 10 monclient: build_initial_monmap 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.396+0000 7fb3f190e640 10 monclient: monmap: 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:53.400 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.396+0000 7fb3f190e640 1 librados: starting msgr at 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.396+0000 7fb3f190e640 1 librados: starting objecter 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 1 Processor -- start 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 1 -- start start 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 1 librados: setting wanted keys 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 1 librados: calling monclient init 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 10 monclient: init 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:53.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:53.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3ec0a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:31:53.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 10 monclient(hunting): picked mon.noname-a con 0x7fb3ec13f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:31:53.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3ec1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:31:53.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec13c190 con 0x7fb3ec13f710 2024-03-25T18:31:53.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec0b7cf0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec13ad50 con 0x7fb3ec0a98f0 2024-03-25T18:31:53.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:53.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.397+0000 7fb3f190e640 10 monclient(hunting): authenticate will time out at 7517.656294s 2024-03-25T18:31:53.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.406+0000 7fb3e3fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fb3ec13f710 0x7fb3ec1df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:38802/0 (socket says 172.21.15.55:38802) 2024-03-25T18:31:53.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.407+0000 7fb3e3fff640 1 -- 172.21.15.55:0/3571174834 learned_addr learned my addr 172.21.15.55:0/3571174834 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:31:53.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.411+0000 7fb3e8ff9640 10 client.?.objecter ms_handle_connect 0x7fb3ec1430c0 2024-03-25T18:31:53.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.411+0000 7fb3e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:53.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.411+0000 7fb3e8ff9640 10 client.?.objecter ms_handle_connect 0x7fb3ec13f710 2024-03-25T18:31:53.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.411+0000 7fb3e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:53.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.412+0000 7fb3e8ff9640 10 client.?.objecter ms_handle_connect 0x7fb3ec0a98f0 2024-03-25T18:31:53.414 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.412+0000 7fb3e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:53.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.420+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639753273 0 0) 0x7fb3ec0b7cf0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.420+0000 7fb3e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:53.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.420+0000 7fb3e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:53.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.420+0000 7fb3e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:53.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.420+0000 7fb3e8ff9640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:31:53.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.420+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3cc004eb0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.425+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896594435 0 0) 0x7fb3ec0b28a0 con 0x7fb3ec13f710 2024-03-25T18:31:53.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.425+0000 7fb3e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:53.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.425+0000 7fb3e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:53.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.425+0000 7fb3e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:53.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.425+0000 7fb3e8ff9640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:31:53.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.426+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ec0b7cf0 con 0x7fb3ec13f710 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366336709 0 0) 0x7fb3ec13ad50 con 0x7fb3ec0a98f0 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ec0b28a0 con 0x7fb3ec0a98f0 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1810950225 0 0) 0x7fb3ec0b7cf0 con 0x7fb3ec13f710 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3ec13ad50 con 0x7fb3ec13f710 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb3d4001850 con 0x7fb3ec13f710 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:53.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 10 monclient(hunting): dump: 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:53.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.439+0000 7fb3e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.441+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1168679398 0 0) 0x7fb3cc004eb0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.441+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3ec0b7cf0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.441+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb3e4001f20 con 0x7fb3ec1430c0 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.441+0000 7fb3e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.441+0000 7fb3e8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.441+0000 7fb3e8ff9640 10 monclient(hunting): dump: 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:53.444 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:53.445 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:53.445 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:53.445 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:53.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 219206476 0 0) 0x7fb3ec0b7cf0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 >> v1:172.21.15.55:6790/0 conn(0x7fb3ec0a98f0 legacy=0x7fb3ec13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:53.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 >> v1:172.21.15.55:6789/0 conn(0x7fb3ec13f710 legacy=0x7fb3ec1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:53.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:31:53.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 1 monclient: found mon.b 2024-03-25T18:31:53.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:53.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:53.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec13afc0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3f190e640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3f190e640 10 monclient: _renew_subs 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3f190e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec13b2a0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3f190e640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3f190e640 10 monclient: _renew_subs 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3f190e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec12fa90 con 0x7fb3ec1430c0 2024-03-25T18:31:53.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.457+0000 7fb3f190e640 1 librados: init done 2024-03-25T18:31:53.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.458+0000 7fb3f190e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:53.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.458+0000 7fb3f190e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:53.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.458+0000 7fb3f190e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:53.461 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.458+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec0b3ef0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.745+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3e40020c0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.745+0000 7fb3e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.745+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb3e4003dc0 con 0x7fb3ec1430c0 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.745+0000 7fb3e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.745+0000 7fb3e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.745+0000 7fb3e8ff9640 10 monclient: dump: 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.747 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:31:53.748 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:53.748 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:31:53.748 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:31:53.748 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:31:53.748 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:31:53.748 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:31:53.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.745+0000 7fb3e9ffb640 4 set_mon_vals no callback set 2024-03-25T18:31:53.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.745+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fb3e4019d90 con 0x7fb3ec1430c0 2024-03-25T18:31:53.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.746+0000 7fb3eaffd640 10 monclient: get_auth_request con 0x7fb3cc06df60 auth_method 0 2024-03-25T18:31:53.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.746+0000 7fb3e8ff9640 10 client.4384.objecter ms_handle_connect 0x7fb3cc06df60 2024-03-25T18:31:53.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.749+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fb3e4053920 con 0x7fb3ec1430c0 2024-03-25T18:31:53.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.749+0000 7fb3e8ff9640 10 client.4384.objecter ms_dispatch 0x7fb3ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:53.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.749+0000 7fb3e8ff9640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:53.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.749+0000 7fb3e8ff9640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:53.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.749+0000 7fb3e8ff9640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T18:31:53.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.751+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3e404f430 con 0x7fb3ec1430c0 2024-03-25T18:31:53.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.751+0000 7fb3e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:53.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:53.751+0000 7fb3e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:54.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.012+0000 7fb3f190e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:54.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.012+0000 7fb3f190e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:54.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.012+0000 7fb3f190e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:31:54.014 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.012+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3ec0627b0 con 0x7fb3ec1430c0 2024-03-25T18:31:54.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.034+0000 7fb3e8ff9640 1 -- 172.21.15.55:0/3571174834 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fb3e4051360 con 0x7fb3ec1430c0 2024-03-25T18:31:54.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.034+0000 7fb3e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:54.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.034+0000 7fb3e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:54.036 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:31:54.036 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:54.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 10 librados: watch_flush enter 2024-03-25T18:31:54.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 10 librados: watch_flush exit 2024-03-25T18:31:54.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T18:31:54.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 10 client.4384.objecter successfully canceled tick 2024-03-25T18:31:54.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 >> v1:172.21.15.55:6800/286536064 conn(0x7fb3cc06df60 legacy=0x7fb3cc070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:54.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 10 monclient: shutdown 2024-03-25T18:31:54.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:54.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 >> v1:172.21.15.133:6789/0 conn(0x7fb3ec1430c0 legacy=0x7fb3ec1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:54.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 shutdown_connections 2024-03-25T18:31:54.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 >> 172.21.15.55:0/3571174834 conn(0x7fb3ec0b7ea0 msgr2=0x7fb3ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:54.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 shutdown_connections 2024-03-25T18:31:54.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 1 -- 172.21.15.55:0/3571174834 wait complete. 2024-03-25T18:31:54.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:31:54.036+0000 7fb3f190e640 1 librados: shutdown 2024-03-25T18:31:54.050 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:54.971 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:04:05.141 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:54.982 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:04:05.152 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:55.014 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:04:05.184 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:55.030 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:04:05.167 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:55.035 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:04:05.172 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:55.061 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:04:05.198 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:55.065 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:04:05.235 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:55.073 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:04:05.243 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:55.101 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:04:05.271 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:00.857 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:00.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.954+0000 7ff5934d6640 10 monclient: get_monmap_and_config 2024-03-25T18:32:00.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.954+0000 7ff5934d6640 10 monclient: build_initial_monmap 2024-03-25T18:32:00.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.954+0000 7ff5934d6640 10 monclient: monmap: 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:32:00.955411+0000 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:32:00.955411+0000 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:32:00.956 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.955+0000 7ff5934d6640 1 Processor -- start 2024-03-25T18:32:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.955+0000 7ff5934d6640 1 -- start start 2024-03-25T18:32:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.955+0000 7ff5934d6640 10 monclient: init 2024-03-25T18:32:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff58c0a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff58c13f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff58c1430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c142520 con 0x7ff58c13f710 2024-03-25T18:32:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c0b28a0 con 0x7ff58c0a98f0 2024-03-25T18:32:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c0b2a30 con 0x7ff58c1430c0 2024-03-25T18:32:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.956+0000 7ff5934d6640 10 monclient(hunting): authenticate will time out at 7525.213634s 2024-03-25T18:32:00.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.969+0000 7ff590a71640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ff58c13f710 0x7ff58c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47382/0 (socket says 172.21.15.55:47382) 2024-03-25T18:32:00.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.969+0000 7ff590a71640 1 -- 172.21.15.55:0/4105483458 learned_addr learned my addr 172.21.15.55:0/4105483458 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:00.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.994+0000 7ff583fff640 1 -- 172.21.15.55:0/4105483458 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470379960 0 0) 0x7ff58c142520 con 0x7ff58c13f710 2024-03-25T18:32:00.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.994+0000 7ff583fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:00.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.994+0000 7ff583fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:00.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.994+0000 7ff583fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:00.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.994+0000 7ff583fff640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:32:00.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:00.994+0000 7ff583fff640 1 -- 172.21.15.55:0/4105483458 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff578003520 con 0x7ff58c13f710 2024-03-25T18:32:01.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 1 -- 172.21.15.55:0/4105483458 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 754747699 0 0) 0x7ff578003520 con 0x7ff58c13f710 2024-03-25T18:32:01.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 1 -- 172.21.15.55:0/4105483458 >> v1:172.21.15.55:6790/0 conn(0x7ff58c1430c0 legacy=0x7ff58c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:01.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 1 -- 172.21.15.55:0/4105483458 >> v1:172.21.15.133:6789/0 conn(0x7ff58c0a98f0 legacy=0x7ff58c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:01.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:01.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 1 monclient: found mon.noname-a 2024-03-25T18:32:01.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 1 -- 172.21.15.55:0/4105483458 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c0b44c0 con 0x7ff58c13f710 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 10 monclient: _finish_auth 0 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 1 -- 172.21.15.55:0/4105483458 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff57c001690 con 0x7ff58c13f710 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff583fff640 10 monclient: dump: 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.009 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:01.010 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:01.010 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:01.010 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:01.010 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:01.010 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:01.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff5934d6640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:32:01.010 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.007+0000 7ff5934d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:01.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.023+0000 7ff583fff640 1 -- 172.21.15.55:0/4105483458 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff57c0017f0 con 0x7ff58c13f710 2024-03-25T18:32:01.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.023+0000 7ff583fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:01.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.023+0000 7ff583fff640 1 -- 172.21.15.55:0/4105483458 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff57c001c40 con 0x7ff58c13f710 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.023+0000 7ff5934d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.023+0000 7ff5934d6640 4 set_mon_vals no callback set 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.023+0000 7ff583fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.023+0000 7ff583fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.023+0000 7ff583fff640 10 monclient: dump: 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:01.025 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:01.026 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.024+0000 7ff5934d6640 10 monclient: shutdown 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.024+0000 7ff5934d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.024+0000 7ff5934d6640 1 -- 172.21.15.55:0/4105483458 >> v1:172.21.15.55:6789/0 conn(0x7ff58c13f710 legacy=0x7ff58c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.024+0000 7ff5934d6640 1 -- 172.21.15.55:0/4105483458 shutdown_connections 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.024+0000 7ff5934d6640 1 -- 172.21.15.55:0/4105483458 >> 172.21.15.55:0/4105483458 conn(0x7ff58c0b7ea0 msgr2=0x7ff58c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.024+0000 7ff5934d6640 1 -- 172.21.15.55:0/4105483458 shutdown_connections 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.024+0000 7ff5934d6640 1 -- 172.21.15.55:0/4105483458 wait complete. 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.025+0000 7ff5934d6640 10 monclient: build_initial_monmap 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.025+0000 7ff5934d6640 10 monclient: monmap: 2024-03-25T18:32:01.028 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.025+0000 7ff5934d6640 1 librados: starting msgr at 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.025+0000 7ff5934d6640 1 librados: starting objecter 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 1 Processor -- start 2024-03-25T18:32:01.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 1 -- start start 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 1 librados: setting wanted keys 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 1 librados: calling monclient init 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 10 monclient: init 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff58c0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff58c13f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff58c1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c138980 con 0x7ff58c0a98f0 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c138640 con 0x7ff58c1430c0 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c0b1700 con 0x7ff58c13f710 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:01.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.026+0000 7ff5934d6640 10 monclient(hunting): authenticate will time out at 7525.284305s 2024-03-25T18:32:01.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.037+0000 7ff591272640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ff58c0a98f0 0x7ff58c137960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47446/0 (socket says 172.21.15.55:47446) 2024-03-25T18:32:01.038 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.037+0000 7ff591272640 1 -- 172.21.15.55:0/1582288771 learned_addr learned my addr 172.21.15.55:0/1582288771 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:01.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.047+0000 7ff581ffb640 10 client.?.objecter ms_handle_connect 0x7ff58c0a98f0 2024-03-25T18:32:01.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.047+0000 7ff581ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:01.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.050+0000 7ff581ffb640 10 client.?.objecter ms_handle_connect 0x7ff58c1430c0 2024-03-25T18:32:01.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.050+0000 7ff581ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:01.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.054+0000 7ff581ffb640 10 client.?.objecter ms_handle_connect 0x7ff58c13f710 2024-03-25T18:32:01.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.055+0000 7ff581ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:01.063 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.062+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963059827 0 0) 0x7ff58c138980 con 0x7ff58c0a98f0 2024-03-25T18:32:01.063 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.062+0000 7ff581ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:01.063 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.062+0000 7ff581ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:01.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.062+0000 7ff581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:01.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.062+0000 7ff581ffb640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:32:01.064 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.062+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff570004eb0 con 0x7ff58c0a98f0 2024-03-25T18:32:01.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.069+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864523685 0 0) 0x7ff58c138640 con 0x7ff58c1430c0 2024-03-25T18:32:01.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.069+0000 7ff581ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:01.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.069+0000 7ff581ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:01.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.069+0000 7ff581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:01.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.069+0000 7ff581ffb640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:32:01.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.069+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff58c138980 con 0x7ff58c1430c0 2024-03-25T18:32:01.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.074+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095283260 0 0) 0x7ff58c0b1700 con 0x7ff58c13f710 2024-03-25T18:32:01.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.074+0000 7ff581ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:01.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.074+0000 7ff581ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:01.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.074+0000 7ff581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:01.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.074+0000 7ff581ffb640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:32:01.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.074+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff58c138640 con 0x7ff58c13f710 2024-03-25T18:32:01.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.080+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2044660455 0 0) 0x7ff570004eb0 con 0x7ff58c0a98f0 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.080+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff58c0b1700 con 0x7ff58c0a98f0 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.080+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff588001a30 con 0x7ff58c0a98f0 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.080+0000 7ff581ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.080+0000 7ff581ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.080+0000 7ff581ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:01.081 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:01.082 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:01.082 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:01.082 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.080+0000 7ff581ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.082+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2622759105 0 0) 0x7ff58c138980 con 0x7ff58c1430c0 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.082+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff570004eb0 con 0x7ff58c1430c0 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.082+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff584001b10 con 0x7ff58c1430c0 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.082+0000 7ff581ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.082+0000 7ff581ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.082+0000 7ff581ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:01.083 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:01.084 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:01.084 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:01.084 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:01.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.089+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 580629955 0 0) 0x7ff58c138640 con 0x7ff58c13f710 2024-03-25T18:32:01.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.089+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff58c138980 con 0x7ff58c13f710 2024-03-25T18:32:01.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.089+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff57c001960 con 0x7ff58c13f710 2024-03-25T18:32:01.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.089+0000 7ff581ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:01.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.089+0000 7ff581ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.089+0000 7ff581ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:01.091 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:01.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 21771611 0 0) 0x7ff58c138980 con 0x7ff58c13f710 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 >> v1:172.21.15.133:6789/0 conn(0x7ff58c1430c0 legacy=0x7ff58c1db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 >> v1:172.21.15.55:6789/0 conn(0x7ff58c0a98f0 legacy=0x7ff58c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 1 monclient: found mon.c 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c0b1290 con 0x7ff58c13f710 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff581ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff5934d6640 5 monclient: authenticate success, global_id 4400 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff5934d6640 10 monclient: _renew_subs 2024-03-25T18:32:01.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff5934d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:32:01.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.097+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58c1e2eb0 con 0x7ff58c13f710 2024-03-25T18:32:01.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.098+0000 7ff5934d6640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:01.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.098+0000 7ff5934d6640 10 monclient: _renew_subs 2024-03-25T18:32:01.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.098+0000 7ff5934d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:32:01.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.098+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c1e3220 con 0x7ff58c13f710 2024-03-25T18:32:01.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.098+0000 7ff5934d6640 1 librados: init done 2024-03-25T18:32:01.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.099+0000 7ff5934d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:01.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.099+0000 7ff5934d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:01.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.099+0000 7ff5934d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:32:01.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.099+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58c0abb30 con 0x7ff58c13f710 2024-03-25T18:32:01.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.417+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff57c004940 con 0x7ff58c13f710 2024-03-25T18:32:01.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.417+0000 7ff581ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:01.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.417+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff57c004c50 con 0x7ff58c13f710 2024-03-25T18:32:01.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.417+0000 7ff581ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:01.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.417+0000 7ff581ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:01.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.417+0000 7ff581ffb640 10 monclient: dump: 2024-03-25T18:32:01.418 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:01.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.417+0000 7ff5837fe640 4 set_mon_vals no callback set 2024-03-25T18:32:01.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.418+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7ff57c01ac20 con 0x7ff58c13f710 2024-03-25T18:32:01.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.418+0000 7ff591272640 10 monclient: get_auth_request con 0x7ff57006e300 auth_method 0 2024-03-25T18:32:01.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.419+0000 7ff581ffb640 10 client.4400.objecter ms_handle_connect 0x7ff57006e300 2024-03-25T18:32:01.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.425+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7ff57c054750 con 0x7ff58c13f710 2024-03-25T18:32:01.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.425+0000 7ff581ffb640 10 client.4400.objecter ms_dispatch 0x7ff58c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:01.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.425+0000 7ff581ffb640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:01.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.425+0000 7ff581ffb640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:01.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.425+0000 7ff581ffb640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-25T18:32:01.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.425+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff57c050260 con 0x7ff58c13f710 2024-03-25T18:32:01.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.425+0000 7ff581ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:01.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.425+0000 7ff581ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:01.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.682+0000 7ff5934d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:01.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.682+0000 7ff5934d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:01.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.682+0000 7ff5934d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:32:01.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.682+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff58c0627b0 con 0x7ff58c13f710 2024-03-25T18:32:01.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.707+0000 7ff581ffb640 1 -- 172.21.15.55:0/1582288771 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7ff57c052190 con 0x7ff58c13f710 2024-03-25T18:32:01.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.707+0000 7ff581ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:01.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.707+0000 7ff581ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:01.708 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:32:01.709 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:01.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.709+0000 7ff5934d6640 10 librados: watch_flush enter 2024-03-25T18:32:01.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.709+0000 7ff5934d6640 10 librados: watch_flush exit 2024-03-25T18:32:01.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 10 client.4400.objecter successfully canceled tick 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 >> v1:172.21.15.55:6800/286536064 conn(0x7ff57006e300 legacy=0x7ff5700707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 10 monclient: shutdown 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 >> v1:172.21.15.55:6790/0 conn(0x7ff58c13f710 legacy=0x7ff58c1da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 shutdown_connections 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 >> 172.21.15.55:0/1582288771 conn(0x7ff58c0b7ea0 msgr2=0x7ff58c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 shutdown_connections 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 1 -- 172.21.15.55:0/1582288771 wait complete. 2024-03-25T18:32:01.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:01.710+0000 7ff5934d6640 1 librados: shutdown 2024-03-25T18:32:01.723 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:08.529 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.643+0000 7f76e7f4b640 10 monclient: get_monmap_and_config 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.643+0000 7f76e7f4b640 10 monclient: build_initial_monmap 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.643+0000 7f76e7f4b640 10 monclient: monmap: 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:32:08.644210+0000 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:32:08.644210+0000 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:08.644 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:08.645 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:32:08.645 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:32:08.645 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 1 Processor -- start 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 1 -- start start 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 10 monclient: init 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 10 monclient(hunting): picked mon.noname-c con 0x7f76e00b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 10 monclient(hunting): picked mon.noname-b con 0x7f76e00b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 10 monclient(hunting): picked mon.noname-a con 0x7f76e014be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e00af940 con 0x7f76e014be90 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.644+0000 7f76e7f4b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e00b0a70 con 0x7f76e00b4d60 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.645+0000 7f76e7f4b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e00b0d50 con 0x7f76e00b1730 2024-03-25T18:32:08.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.645+0000 7f76e7f4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:08.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.645+0000 7f76e7f4b640 10 monclient(hunting): authenticate will time out at 7532.902368s 2024-03-25T18:32:08.659 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.658+0000 7f76e54e6640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f76e00b1730 0x7f76e00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:38310/0 (socket says 172.21.15.55:38310) 2024-03-25T18:32:08.659 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.658+0000 7f76e54e6640 1 -- 172.21.15.55:0/3837138340 learned_addr learned my addr 172.21.15.55:0/3837138340 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:08.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.673+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512578531 0 0) 0x7f76e00b0a70 con 0x7f76e00b4d60 2024-03-25T18:32:08.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.673+0000 7f76e4ce5640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.673+0000 7f76e4ce5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.673+0000 7f76e4ce5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.673+0000 7f76e4ce5640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:32:08.674 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.673+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc003520 con 0x7f76e00b4d60 2024-03-25T18:32:08.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055487027 0 0) 0x7f76e00b0d50 con 0x7f76e00b1730 2024-03-25T18:32:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:32:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e00b0a70 con 0x7f76e00b1730 2024-03-25T18:32:08.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253410651 0 0) 0x7f76e00af940 con 0x7f76e014be90 2024-03-25T18:32:08.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:32:08.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.677+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e00b0d50 con 0x7f76e014be90 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899093172 0 0) 0x7f76cc003520 con 0x7f76e00b4d60 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 >> v1:172.21.15.55:6790/0 conn(0x7f76e00b1730 legacy=0x7f76e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 >> v1:172.21.15.55:6789/0 conn(0x7f76e014be90 legacy=0x7f76e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 10 monclient: _finish_hunting 0 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 1 monclient: found mon.noname-c 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e00b1030 con 0x7f76e00b4d60 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 10 monclient: _finish_auth 0 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 10 monclient: _check_auth_tickets 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:08.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f76dc001690 con 0x7f76e00b4d60 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e4ce5640 10 monclient: dump: 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e7f4b640 5 monclient: authenticate success, global_id 4393 2024-03-25T18:32:08.690 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.688+0000 7f76e7f4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:08.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.703+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76dc0017f0 con 0x7f76e00b4d60 2024-03-25T18:32:08.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.703+0000 7f76e4ce5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:08.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.703+0000 7f76e4ce5640 1 -- 172.21.15.55:0/3837138340 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f76dc001c00 con 0x7f76e00b4d60 2024-03-25T18:32:08.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.703+0000 7f76e4ce5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:08.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.703+0000 7f76e4ce5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:08.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.703+0000 7f76e4ce5640 10 monclient: dump: 2024-03-25T18:32:08.704 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.703+0000 7f76e7f4b640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:08.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.703+0000 7f76e7f4b640 4 set_mon_vals no callback set 2024-03-25T18:32:08.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.704+0000 7f76e7f4b640 10 monclient: shutdown 2024-03-25T18:32:08.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.704+0000 7f76e7f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:08.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.704+0000 7f76e7f4b640 1 -- 172.21.15.55:0/3837138340 >> v1:172.21.15.133:6789/0 conn(0x7f76e00b4d60 legacy=0x7f76e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.704+0000 7f76e7f4b640 1 -- 172.21.15.55:0/3837138340 shutdown_connections 2024-03-25T18:32:08.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.704+0000 7f76e7f4b640 1 -- 172.21.15.55:0/3837138340 >> 172.21.15.55:0/3837138340 conn(0x7f76e00617a0 msgr2=0x7f76e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:08.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.705+0000 7f76e7f4b640 1 -- 172.21.15.55:0/3837138340 shutdown_connections 2024-03-25T18:32:08.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.705+0000 7f76e7f4b640 1 -- 172.21.15.55:0/3837138340 wait complete. 2024-03-25T18:32:08.706 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.705+0000 7f76e7f4b640 10 monclient: build_initial_monmap 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.705+0000 7f76e7f4b640 10 monclient: monmap: 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:32:08.707 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:08.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 1 librados: starting msgr at 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 1 librados: starting objecter 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 1 Processor -- start 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 1 -- start start 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 1 librados: setting wanted keys 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 1 librados: calling monclient init 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 10 monclient: init 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 10 monclient(hunting): picked mon.noname-a con 0x7f76e00b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 10 monclient(hunting): picked mon.noname-b con 0x7f76e00b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 10 monclient(hunting): picked mon.noname-c con 0x7f76e014be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e0140c00 con 0x7f76e00b1730 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e0138880 con 0x7f76e00b4d60 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e0138a10 con 0x7f76e014be90 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:08.709 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.706+0000 7f76e7f4b640 10 monclient(hunting): authenticate will time out at 7532.964280s 2024-03-25T18:32:08.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.714+0000 7f76e5ce7640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f76e00b1730 0x7f76e01429c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:40974/0 (socket says 172.21.15.55:40974) 2024-03-25T18:32:08.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.714+0000 7f76e5ce7640 1 -- 172.21.15.55:0/1460704779 learned_addr learned my addr 172.21.15.55:0/1460704779 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:08.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.718+0000 7f76d67fc640 10 client.?.objecter ms_handle_connect 0x7f76e00b1730 2024-03-25T18:32:08.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.718+0000 7f76d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:08.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.723+0000 7f76d67fc640 10 client.?.objecter ms_handle_connect 0x7f76e014be90 2024-03-25T18:32:08.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.723+0000 7f76d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:08.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.729+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467807662 0 0) 0x7f76e0140c00 con 0x7f76e00b1730 2024-03-25T18:32:08.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.729+0000 7f76d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.729+0000 7f76d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.729+0000 7f76d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.729+0000 7f76d67fc640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:32:08.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.729+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c4004eb0 con 0x7f76e00b1730 2024-03-25T18:32:08.734 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.734+0000 7f76d67fc640 10 client.?.objecter ms_handle_connect 0x7f76e00b4d60 2024-03-25T18:32:08.734 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.734+0000 7f76d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:08.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.745+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4225534557 0 0) 0x7f76c4004eb0 con 0x7f76e00b1730 2024-03-25T18:32:08.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.745+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e0140c00 con 0x7f76e00b1730 2024-03-25T18:32:08.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.745+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f76dc001b80 con 0x7f76e00b1730 2024-03-25T18:32:08.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.745+0000 7f76d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:08.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.745+0000 7f76d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:08.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.745+0000 7f76d67fc640 10 monclient(hunting): dump: 2024-03-25T18:32:08.746 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:08.746 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:08.747 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:08.747 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:08.747 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:08.747 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:08.747 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:08.747 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:08.747 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:08.747 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:08.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.745+0000 7f76d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.746+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473099374 0 0) 0x7f76e0138a10 con 0x7f76e014be90 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.746+0000 7f76d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.746+0000 7f76d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.746+0000 7f76d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.746+0000 7f76d67fc640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:32:08.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.746+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c4004eb0 con 0x7f76e014be90 2024-03-25T18:32:08.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.749+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563746169 0 0) 0x7f76e0138880 con 0x7f76e00b4d60 2024-03-25T18:32:08.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.749+0000 7f76d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:08.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.749+0000 7f76d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:08.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.749+0000 7f76d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:08.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.749+0000 7f76d67fc640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:32:08.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.749+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e0138a10 con 0x7f76e00b4d60 2024-03-25T18:32:08.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1559420049 0 0) 0x7f76e0140c00 con 0x7f76e00b1730 2024-03-25T18:32:08.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 >> v1:172.21.15.55:6790/0 conn(0x7f76e014be90 legacy=0x7f76e01db790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 >> v1:172.21.15.133:6789/0 conn(0x7f76e00b4d60 legacy=0x7f76e01db080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:08.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:08.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 1 monclient: found mon.a 2024-03-25T18:32:08.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:08.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:08.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e0138ba0 con 0x7f76e00b1730 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.756+0000 7f76e7f4b640 5 monclient: authenticate success, global_id 4413 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 10 monclient: _renew_subs 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e01421b0 con 0x7f76e00b1730 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 10 monclient: _renew_subs 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e01430d0 con 0x7f76e00b1730 2024-03-25T18:32:08.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 1 librados: init done 2024-03-25T18:32:08.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:08.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:08.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:08.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:08.757+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e00627b0 con 0x7f76e00b1730 2024-03-25T18:32:09.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.005+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76dc0038a0 con 0x7f76e00b1730 2024-03-25T18:32:09.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.005+0000 7f76d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:09.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.005+0000 7f76d7fff640 4 set_mon_vals no callback set 2024-03-25T18:32:09.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.072+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f76dc003c70 con 0x7f76e00b1730 2024-03-25T18:32:09.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.072+0000 7f76d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:09.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.072+0000 7f76d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.072+0000 7f76d67fc640 10 monclient: dump: 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:09.074 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:09.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.073+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f76dc003e50 con 0x7f76e00b1730 2024-03-25T18:32:09.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.074+0000 7f76e54e6640 10 monclient: get_auth_request con 0x7f76c406de10 auth_method 0 2024-03-25T18:32:09.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.074+0000 7f76d67fc640 10 client.4413.objecter ms_handle_connect 0x7f76c406de10 2024-03-25T18:32:09.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.082+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f76dc054940 con 0x7f76e00b1730 2024-03-25T18:32:09.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.082+0000 7f76d67fc640 10 client.4413.objecter ms_dispatch 0x7f76e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:09.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.082+0000 7f76d67fc640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:09.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.082+0000 7f76d67fc640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:09.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.082+0000 7f76d67fc640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-25T18:32:09.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.082+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76dc050450 con 0x7f76e00b1730 2024-03-25T18:32:09.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.082+0000 7f76d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:09.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.082+0000 7f76d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:09.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.339+0000 7f76e7f4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:09.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.339+0000 7f76e7f4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:09.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.339+0000 7f76e7f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:09.340 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.339+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76e0069f40 con 0x7f76e00b1730 2024-03-25T18:32:09.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.369+0000 7f76d67fc640 1 -- 172.21.15.55:0/1460704779 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f76dc052380 con 0x7f76e00b1730 2024-03-25T18:32:09.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.369+0000 7f76d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:09.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.369+0000 7f76d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:09.370 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:32:09.370 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:09.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 10 librados: watch_flush enter 2024-03-25T18:32:09.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 10 librados: watch_flush exit 2024-03-25T18:32:09.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-25T18:32:09.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 10 client.4413.objecter successfully canceled tick 2024-03-25T18:32:09.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 >> v1:172.21.15.55:6800/286536064 conn(0x7f76c406de10 legacy=0x7f76c40702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:09.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 10 monclient: shutdown 2024-03-25T18:32:09.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:09.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 >> v1:172.21.15.55:6789/0 conn(0x7f76e00b1730 legacy=0x7f76e01429c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:09.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 shutdown_connections 2024-03-25T18:32:09.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 >> 172.21.15.55:0/1460704779 conn(0x7f76e00617a0 msgr2=0x7f76e00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:09.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 shutdown_connections 2024-03-25T18:32:09.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 1 -- 172.21.15.55:0/1460704779 wait complete. 2024-03-25T18:32:09.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:09.371+0000 7f76e7f4b640 1 librados: shutdown 2024-03-25T18:32:09.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:16.189 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.288+0000 7f5199a25640 10 monclient: get_monmap_and_config 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.288+0000 7f5199a25640 10 monclient: build_initial_monmap 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.288+0000 7f5199a25640 10 monclient: monmap: 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:32:16.289774+0000 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:32:16.289774+0000 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:32:16.290 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:32:16.291 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:16.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.289+0000 7f5199a25640 1 Processor -- start 2024-03-25T18:32:16.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.289+0000 7f5199a25640 1 -- start start 2024-03-25T18:32:16.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.289+0000 7f5199a25640 10 monclient: init 2024-03-25T18:32:16.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:16.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:16.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 10 monclient(hunting): picked mon.noname-b con 0x7f51940a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:16.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 10 monclient(hunting): picked mon.noname-c con 0x7f519413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:16.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 10 monclient(hunting): picked mon.noname-a con 0x7f51941430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:16.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5194142520 con 0x7f51941430c0 2024-03-25T18:32:16.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51940b28a0 con 0x7f519413f710 2024-03-25T18:32:16.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51940b2a30 con 0x7f51940a98f0 2024-03-25T18:32:16.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:16.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.290+0000 7f5199a25640 10 monclient(hunting): authenticate will time out at 7540.548266s 2024-03-25T18:32:16.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.293+0000 7f5192ffd640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f51940a98f0 0x7f51940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:42792/0 (socket says 172.21.15.55:42792) 2024-03-25T18:32:16.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.293+0000 7f5192ffd640 1 -- 172.21.15.55:0/3790804220 learned_addr learned my addr 172.21.15.55:0/3790804220 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:16.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.328+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799726669 0 0) 0x7f5194142520 con 0x7f51941430c0 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.328+0000 7f51927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.328+0000 7f51927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.328+0000 7f51927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.328+0000 7f51927fc640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.328+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5180003520 con 0x7f51941430c0 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.329+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650652279 0 0) 0x7f51940b28a0 con 0x7f519413f710 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.329+0000 7f51927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.329+0000 7f51927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.329+0000 7f51927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.329+0000 7f51927fc640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:32:16.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.329+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5194142520 con 0x7f519413f710 2024-03-25T18:32:16.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 564580489 0 0) 0x7f5194142520 con 0x7f519413f710 2024-03-25T18:32:16.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 >> v1:172.21.15.55:6790/0 conn(0x7f51940a98f0 legacy=0x7f51940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:16.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 >> v1:172.21.15.55:6789/0 conn(0x7f51941430c0 legacy=0x7f519414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:16.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 1 monclient: found mon.noname-c 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51940b44c0 con 0x7f519413f710 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f517c001670 con 0x7f519413f710 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f5199a25640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f5199a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f51927fc640 10 monclient: dump: 2024-03-25T18:32:16.348 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:16.349 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.346+0000 7f5199a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:16.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.356+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f517c0017d0 con 0x7f519413f710 2024-03-25T18:32:16.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.356+0000 7f51927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:16.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.356+0000 7f51927fc640 1 -- 172.21.15.55:0/3790804220 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f517c001c20 con 0x7f519413f710 2024-03-25T18:32:16.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.356+0000 7f51927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:16.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.356+0000 7f51927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:16.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.356+0000 7f51927fc640 10 monclient: dump: 2024-03-25T18:32:16.357 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:16.357 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.356+0000 7f5199a25640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:16.358 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.356+0000 7f5199a25640 4 set_mon_vals no callback set 2024-03-25T18:32:16.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.357+0000 7f5199a25640 10 monclient: shutdown 2024-03-25T18:32:16.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.357+0000 7f5199a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:16.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.357+0000 7f5199a25640 1 -- 172.21.15.55:0/3790804220 >> v1:172.21.15.133:6789/0 conn(0x7f519413f710 legacy=0x7f51941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:16.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.357+0000 7f5199a25640 1 -- 172.21.15.55:0/3790804220 shutdown_connections 2024-03-25T18:32:16.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.357+0000 7f5199a25640 1 -- 172.21.15.55:0/3790804220 >> 172.21.15.55:0/3790804220 conn(0x7f51940b7ea0 msgr2=0x7f51940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:16.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.357+0000 7f5199a25640 1 -- 172.21.15.55:0/3790804220 shutdown_connections 2024-03-25T18:32:16.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.357+0000 7f5199a25640 1 -- 172.21.15.55:0/3790804220 wait complete. 2024-03-25T18:32:16.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.358+0000 7f5199a25640 10 monclient: build_initial_monmap 2024-03-25T18:32:16.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.358+0000 7f5199a25640 10 monclient: monmap: 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.358+0000 7f5199a25640 1 librados: starting msgr at 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.358+0000 7f5199a25640 1 librados: starting objecter 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.358+0000 7f5199a25640 1 Processor -- start 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.358+0000 7f5199a25640 1 -- start start 2024-03-25T18:32:16.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.358+0000 7f5199a25640 1 librados: setting wanted keys 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.358+0000 7f5199a25640 1 librados: calling monclient init 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.358+0000 7f5199a25640 10 monclient: init 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 10 monclient(hunting): picked mon.noname-c con 0x7f51940a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 10 monclient(hunting): picked mon.noname-b con 0x7f519413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 10 monclient(hunting): picked mon.noname-a con 0x7f51941430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519413b260 con 0x7f51941430c0 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519413b540 con 0x7f519413f710 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519413a100 con 0x7f51940a98f0 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:16.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.359+0000 7f5199a25640 10 monclient(hunting): authenticate will time out at 7540.616444s 2024-03-25T18:32:16.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.370+0000 7f518bfff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f519413f710 0x7f51941df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:59322/0 (socket says 172.21.15.55:59322) 2024-03-25T18:32:16.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.370+0000 7f518bfff640 1 -- 172.21.15.55:0/3120115738 learned_addr learned my addr 172.21.15.55:0/3120115738 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:16.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.370+0000 7f5190ff9640 10 client.?.objecter ms_handle_connect 0x7f519413f710 2024-03-25T18:32:16.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.370+0000 7f5190ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:16.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.377+0000 7f5190ff9640 10 client.?.objecter ms_handle_connect 0x7f51941430c0 2024-03-25T18:32:16.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.377+0000 7f5190ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:16.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.379+0000 7f5190ff9640 10 client.?.objecter ms_handle_connect 0x7f51940a98f0 2024-03-25T18:32:16.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.379+0000 7f5190ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:16.385 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.385+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865809433 0 0) 0x7f519413b540 con 0x7f519413f710 2024-03-25T18:32:16.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.385+0000 7f5190ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:16.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.385+0000 7f5190ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:16.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.385+0000 7f5190ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:16.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.385+0000 7f5190ff9640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:32:16.386 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.385+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5174004eb0 con 0x7f519413f710 2024-03-25T18:32:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.387+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480120131 0 0) 0x7f519413a100 con 0x7f51940a98f0 2024-03-25T18:32:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.387+0000 7f5190ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.387+0000 7f5190ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.387+0000 7f5190ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.387+0000 7f5190ff9640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:32:16.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.387+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f519413b540 con 0x7f51940a98f0 2024-03-25T18:32:16.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.390+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454838044 0 0) 0x7f519413b260 con 0x7f51941430c0 2024-03-25T18:32:16.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.390+0000 7f5190ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:16.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.390+0000 7f5190ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:16.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.390+0000 7f5190ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:16.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.390+0000 7f5190ff9640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:32:16.391 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.390+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f519413a100 con 0x7f51941430c0 2024-03-25T18:32:16.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.398+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070410590 0 0) 0x7f5174004eb0 con 0x7f519413f710 2024-03-25T18:32:16.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.398+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f519413b260 con 0x7f519413f710 2024-03-25T18:32:16.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.398+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f517c001630 con 0x7f519413f710 2024-03-25T18:32:16.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.398+0000 7f5190ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:16.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.398+0000 7f5190ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:16.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.398+0000 7f5190ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:16.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.398+0000 7f5190ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:16.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.405+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 831209357 0 0) 0x7f519413a100 con 0x7f51941430c0 2024-03-25T18:32:16.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.405+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5174004eb0 con 0x7f51941430c0 2024-03-25T18:32:16.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.405+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f518c001c10 con 0x7f51941430c0 2024-03-25T18:32:16.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.405+0000 7f5190ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:16.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.405+0000 7f5190ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:16.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.405+0000 7f5190ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:16.406 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:16.406 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:16.406 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.407 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.407 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:16.407 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:16.407 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:16.407 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:16.407 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:16.407 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:16.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.415+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 974344865 0 0) 0x7f519413b540 con 0x7f51940a98f0 2024-03-25T18:32:16.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.415+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f519413a100 con 0x7f51940a98f0 2024-03-25T18:32:16.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.415+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f51840018a0 con 0x7f51940a98f0 2024-03-25T18:32:16.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.415+0000 7f5190ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:16.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.415+0000 7f5190ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:16.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.415+0000 7f5190ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:16.416 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:16.417 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:16.417 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.417 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.417 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:16.417 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:16.417 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:16.417 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:16.417 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:16.417 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:16.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 479657203 0 0) 0x7f5174004eb0 con 0x7f51941430c0 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 >> v1:172.21.15.55:6790/0 conn(0x7f51940a98f0 legacy=0x7f51940b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 >> v1:172.21.15.133:6789/0 conn(0x7f519413f710 legacy=0x7f51941df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 1 monclient: found mon.a 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51940b7770 con 0x7f51941430c0 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 10 monclient: _finish_auth 0 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5190ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5199a25640 5 monclient: authenticate success, global_id 4419 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5199a25640 10 monclient: _renew_subs 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5199a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:16.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5194139d20 con 0x7f51941430c0 2024-03-25T18:32:16.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5199a25640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:16.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5199a25640 10 monclient: _renew_subs 2024-03-25T18:32:16.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5199a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:16.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51940b7ca0 con 0x7f51941430c0 2024-03-25T18:32:16.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.423+0000 7f5199a25640 1 librados: init done 2024-03-25T18:32:16.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.424+0000 7f5199a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:16.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.424+0000 7f5199a25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:16.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.424+0000 7f5199a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:16.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.424+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51940b3ef0 con 0x7f51941430c0 2024-03-25T18:32:16.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.734+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f518c002020 con 0x7f51941430c0 2024-03-25T18:32:16.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.734+0000 7f5190ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:16.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.734+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f518c003990 con 0x7f51941430c0 2024-03-25T18:32:16.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.734+0000 7f5190ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:16.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.734+0000 7f5190ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:16.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.734+0000 7f5190ff9640 10 monclient: dump: 2024-03-25T18:32:16.735 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:16.735 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:16.735 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.736 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:16.736 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:16.736 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:16.736 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:16.736 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:16.736 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:16.736 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:16.736 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.734+0000 7f5191ffb640 4 set_mon_vals no callback set 2024-03-25T18:32:16.737 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.735+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f518c019960 con 0x7f51941430c0 2024-03-25T18:32:16.737 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.735+0000 7f5192ffd640 10 monclient: get_auth_request con 0x7f517406e030 auth_method 0 2024-03-25T18:32:16.737 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.736+0000 7f5190ff9640 10 client.4419.objecter ms_handle_connect 0x7f517406e030 2024-03-25T18:32:16.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.741+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f518c053680 con 0x7f51941430c0 2024-03-25T18:32:16.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.741+0000 7f5190ff9640 10 client.4419.objecter ms_dispatch 0x7f5194000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:16.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.741+0000 7f5190ff9640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:16.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.741+0000 7f5190ff9640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:16.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.741+0000 7f5190ff9640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-25T18:32:16.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.741+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f518c04efc0 con 0x7f51941430c0 2024-03-25T18:32:16.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.741+0000 7f5190ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:16.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.741+0000 7f5190ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:16.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.992+0000 7f5199a25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:16.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.992+0000 7f5199a25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:16.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.992+0000 7f5199a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:16.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:16.992+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51940627b0 con 0x7f51941430c0 2024-03-25T18:32:17.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.035+0000 7f5190ff9640 1 -- 172.21.15.55:0/3120115738 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f518c0510c0 con 0x7f51941430c0 2024-03-25T18:32:17.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.035+0000 7f5190ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:17.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.035+0000 7f5190ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:17.037 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:32:17.037 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:17.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 10 librados: watch_flush enter 2024-03-25T18:32:17.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 10 librados: watch_flush exit 2024-03-25T18:32:17.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-25T18:32:17.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 10 client.4419.objecter successfully canceled tick 2024-03-25T18:32:17.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 >> v1:172.21.15.55:6800/286536064 conn(0x7f517406e030 legacy=0x7f51740704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:17.039 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 10 monclient: shutdown 2024-03-25T18:32:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 >> v1:172.21.15.55:6789/0 conn(0x7f51941430c0 legacy=0x7f51941e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 shutdown_connections 2024-03-25T18:32:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 >> 172.21.15.55:0/3120115738 conn(0x7f51940b7ea0 msgr2=0x7f5194074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 shutdown_connections 2024-03-25T18:32:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 1 -- 172.21.15.55:0/3120115738 wait complete. 2024-03-25T18:32:17.040 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:17.038+0000 7f5199a25640 1 librados: shutdown 2024-03-25T18:32:17.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:23.857 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:23.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.975+0000 7efe9160d640 10 monclient: get_monmap_and_config 2024-03-25T18:32:23.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.975+0000 7efe9160d640 10 monclient: build_initial_monmap 2024-03-25T18:32:23.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.975+0000 7efe9160d640 10 monclient: monmap: 2024-03-25T18:32:23.976 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:23.976 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:23.976 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:32:23.976316+0000 2024-03-25T18:32:23.976 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:32:23.976316+0000 2024-03-25T18:32:23.976 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:23.976 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:23.977 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:23.977 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:32:23.977 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:32:23.977 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.976+0000 7efe9160d640 1 Processor -- start 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.976+0000 7efe9160d640 1 -- start start 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.976+0000 7efe9160d640 10 monclient: init 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.976+0000 7efe9160d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.976+0000 7efe9160d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.976+0000 7efe9160d640 10 monclient(hunting): picked mon.noname-a con 0x7efe8c0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.977+0000 7efe9160d640 10 monclient(hunting): picked mon.noname-c con 0x7efe8c13f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.977+0000 7efe9160d640 10 monclient(hunting): picked mon.noname-b con 0x7efe8c1430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.977+0000 7efe9160d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8c142520 con 0x7efe8c0a98f0 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.977+0000 7efe9160d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8c0b28a0 con 0x7efe8c13f710 2024-03-25T18:32:23.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.977+0000 7efe9160d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8c0b2a30 con 0x7efe8c1430c0 2024-03-25T18:32:23.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.977+0000 7efe9160d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:23.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.977+0000 7efe9160d640 10 monclient(hunting): authenticate will time out at 7548.234429s 2024-03-25T18:32:23.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.989+0000 7efe83fff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7efe8c13f710 0x7efe8c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:52164/0 (socket says 172.21.15.55:52164) 2024-03-25T18:32:23.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:23.989+0000 7efe83fff640 1 -- 172.21.15.55:0/544049513 learned_addr learned my addr 172.21.15.55:0/544049513 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.026+0000 7efe8a7fc640 1 -- 172.21.15.55:0/544049513 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699847954 0 0) 0x7efe8c142520 con 0x7efe8c0a98f0 2024-03-25T18:32:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.026+0000 7efe8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.026+0000 7efe8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.026+0000 7efe8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.026+0000 7efe8a7fc640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:32:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.026+0000 7efe8a7fc640 1 -- 172.21.15.55:0/544049513 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe78003520 con 0x7efe8c0a98f0 2024-03-25T18:32:24.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 1 -- 172.21.15.55:0/544049513 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3637169082 0 0) 0x7efe78003520 con 0x7efe8c0a98f0 2024-03-25T18:32:24.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 1 -- 172.21.15.55:0/544049513 >> v1:172.21.15.55:6790/0 conn(0x7efe8c1430c0 legacy=0x7efe8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:24.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 1 -- 172.21.15.55:0/544049513 >> v1:172.21.15.133:6789/0 conn(0x7efe8c13f710 legacy=0x7efe8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:24.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:24.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:32:24.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:24.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:32:24.106 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 1 -- 172.21.15.55:0/544049513 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c0b44c0 con 0x7efe8c0a98f0 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 1 -- 172.21.15.55:0/544049513 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efe7c001690 con 0x7efe8c0a98f0 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe9160d640 5 monclient: authenticate success, global_id 4422 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe9160d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe8a7fc640 10 monclient: dump: 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:24.107 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:24.108 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:24.108 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:24.108 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:24.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.105+0000 7efe9160d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.109+0000 7efe8a7fc640 1 -- 172.21.15.55:0/544049513 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe7c0017f0 con 0x7efe8c0a98f0 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.109+0000 7efe8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.109+0000 7efe8a7fc640 1 -- 172.21.15.55:0/544049513 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efe7c001c00 con 0x7efe8c0a98f0 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.109+0000 7efe8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.109+0000 7efe8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.109+0000 7efe8a7fc640 10 monclient: dump: 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:24.110 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:24.111 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:24.111 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:24.111 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:24.111 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:24.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.109+0000 7efe9160d640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:24.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.109+0000 7efe9160d640 4 set_mon_vals no callback set 2024-03-25T18:32:24.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.110+0000 7efe9160d640 10 monclient: shutdown 2024-03-25T18:32:24.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.110+0000 7efe9160d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:24.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.110+0000 7efe9160d640 1 -- 172.21.15.55:0/544049513 >> v1:172.21.15.55:6789/0 conn(0x7efe8c0a98f0 legacy=0x7efe8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:24.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.110+0000 7efe9160d640 1 -- 172.21.15.55:0/544049513 shutdown_connections 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.110+0000 7efe9160d640 1 -- 172.21.15.55:0/544049513 >> 172.21.15.55:0/544049513 conn(0x7efe8c0b7ea0 msgr2=0x7efe8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.110+0000 7efe9160d640 1 -- 172.21.15.55:0/544049513 shutdown_connections 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.110+0000 7efe9160d640 1 -- 172.21.15.55:0/544049513 wait complete. 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.111+0000 7efe9160d640 10 monclient: build_initial_monmap 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.111+0000 7efe9160d640 10 monclient: monmap: 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:32:24.113 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:32:24.114 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:24.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.111+0000 7efe9160d640 1 librados: starting msgr at 2024-03-25T18:32:24.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.111+0000 7efe9160d640 1 librados: starting objecter 2024-03-25T18:32:24.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.111+0000 7efe9160d640 1 Processor -- start 2024-03-25T18:32:24.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.111+0000 7efe9160d640 1 -- start start 2024-03-25T18:32:24.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.111+0000 7efe9160d640 1 librados: setting wanted keys 2024-03-25T18:32:24.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.111+0000 7efe9160d640 1 librados: calling monclient init 2024-03-25T18:32:24.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.111+0000 7efe9160d640 10 monclient: init 2024-03-25T18:32:24.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:24.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:24.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 10 monclient(hunting): picked mon.noname-c con 0x7efe8c0a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 10 monclient(hunting): picked mon.noname-b con 0x7efe8c13f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 10 monclient(hunting): picked mon.noname-a con 0x7efe8c1430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8c1d7f30 con 0x7efe8c1430c0 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8c1d80c0 con 0x7efe8c13f710 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8c1d8250 con 0x7efe8c0a98f0 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.112+0000 7efe9160d640 10 monclient(hunting): authenticate will time out at 7548.370086s 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.114+0000 7efe83fff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7efe8c13f710 0x7efe8c1d4550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:52206/0 (socket says 172.21.15.55:52206) 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.114+0000 7efe83fff640 1 -- 172.21.15.55:0/3731353203 learned_addr learned my addr 172.21.15.55:0/3731353203 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.114+0000 7efe837fe640 10 client.?.objecter ms_handle_connect 0x7efe8c13f710 2024-03-25T18:32:24.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.114+0000 7efe837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:24.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.124+0000 7efe837fe640 10 client.?.objecter ms_handle_connect 0x7efe8c0a98f0 2024-03-25T18:32:24.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.124+0000 7efe837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:24.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.125+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285811613 0 0) 0x7efe8c1d80c0 con 0x7efe8c13f710 2024-03-25T18:32:24.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.126+0000 7efe837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:24.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.126+0000 7efe837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:24.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.126+0000 7efe837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:24.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.126+0000 7efe837fe640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:32:24.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.126+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe6c004eb0 con 0x7efe8c13f710 2024-03-25T18:32:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.135+0000 7efe837fe640 10 client.?.objecter ms_handle_connect 0x7efe8c1430c0 2024-03-25T18:32:24.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.135+0000 7efe837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 26100523 0 0) 0x7efe6c004eb0 con 0x7efe8c13f710 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe8c1d80c0 con 0x7efe8c13f710 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efe740018b0 con 0x7efe8c13f710 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 10 monclient(hunting): dump: 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.142 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621401188 0 0) 0x7efe8c1d8250 con 0x7efe8c0a98f0 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:32:24.143 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.141+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe6c004eb0 con 0x7efe8c0a98f0 2024-03-25T18:32:24.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.154+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242215006 0 0) 0x7efe8c1d7f30 con 0x7efe8c1430c0 2024-03-25T18:32:24.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.154+0000 7efe837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:24.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.154+0000 7efe837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:24.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.154+0000 7efe837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:24.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.154+0000 7efe837fe640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:32:24.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.154+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe8c1d8250 con 0x7efe8c1430c0 2024-03-25T18:32:24.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.157+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1329718529 0 0) 0x7efe6c004eb0 con 0x7efe8c0a98f0 2024-03-25T18:32:24.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.157+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe8c1d7f30 con 0x7efe8c0a98f0 2024-03-25T18:32:24.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.157+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efe7c001b00 con 0x7efe8c0a98f0 2024-03-25T18:32:24.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.157+0000 7efe837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:24.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.157+0000 7efe837fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:24.158 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.157+0000 7efe837fe640 10 monclient(hunting): dump: 2024-03-25T18:32:24.158 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:24.158 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:24.159 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.159 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.159 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:24.159 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:24.159 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:24.159 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:24.159 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:24.159 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:24.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3057889601 0 0) 0x7efe8c1d80c0 con 0x7efe8c13f710 2024-03-25T18:32:24.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 >> v1:172.21.15.55:6790/0 conn(0x7efe8c0a98f0 legacy=0x7efe8c1d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:24.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 >> v1:172.21.15.55:6789/0 conn(0x7efe8c1430c0 legacy=0x7efe8c1d4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:24.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:24.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 1 monclient: found mon.b 2024-03-25T18:32:24.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c13b600 con 0x7efe8c13f710 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe9160d640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe9160d640 10 monclient: _renew_subs 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe9160d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8c13b850 con 0x7efe8c13f710 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe9160d640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe9160d640 10 monclient: _renew_subs 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe9160d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe8c1e31c0 con 0x7efe8c13f710 2024-03-25T18:32:24.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.166+0000 7efe9160d640 1 librados: init done 2024-03-25T18:32:24.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.167+0000 7efe9160d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:24.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.167+0000 7efe9160d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:24.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.167+0000 7efe9160d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:32:24.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.167+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8c1d4270 con 0x7efe8c13f710 2024-03-25T18:32:24.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.445+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe740047e0 con 0x7efe8c13f710 2024-03-25T18:32:24.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.445+0000 7efe837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:24.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.445+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efe74004a90 con 0x7efe8c13f710 2024-03-25T18:32:24.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.445+0000 7efe837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:24.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.445+0000 7efe837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:24.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.445+0000 7efe837fe640 10 monclient: dump: 2024-03-25T18:32:24.446 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:24.447 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:24.447 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.447 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:24.447 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:24.447 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:24.447 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:24.447 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:24.447 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:24.449 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:24.449 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.445+0000 7efe89ffb640 4 set_mon_vals no callback set 2024-03-25T18:32:24.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.446+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7efe74004cb0 con 0x7efe8c13f710 2024-03-25T18:32:24.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.446+0000 7efe8affd640 10 monclient: get_auth_request con 0x7efe6c06e020 auth_method 0 2024-03-25T18:32:24.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.446+0000 7efe837fe640 10 client.4405.objecter ms_handle_connect 0x7efe6c06e020 2024-03-25T18:32:24.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.450+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7efe740546b0 con 0x7efe8c13f710 2024-03-25T18:32:24.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.450+0000 7efe837fe640 10 client.4405.objecter ms_dispatch 0x7efe8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:24.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.450+0000 7efe837fe640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:24.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.450+0000 7efe837fe640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:24.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.450+0000 7efe837fe640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T18:32:24.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.451+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe740501c0 con 0x7efe8c13f710 2024-03-25T18:32:24.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.451+0000 7efe837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:24.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.451+0000 7efe837fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:24.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.720+0000 7efe9160d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:24.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.720+0000 7efe9160d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:24.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.720+0000 7efe9160d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:32:24.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.720+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe8c0627b0 con 0x7efe8c13f710 2024-03-25T18:32:24.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.746+0000 7efe837fe640 1 -- 172.21.15.55:0/3731353203 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7efe740520f0 con 0x7efe8c13f710 2024-03-25T18:32:24.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.746+0000 7efe837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:24.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.746+0000 7efe837fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:24.747 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:32:24.747 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 10 librados: watch_flush enter 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 10 librados: watch_flush exit 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 10 client.4405.objecter successfully canceled tick 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 >> v1:172.21.15.55:6800/286536064 conn(0x7efe6c06e020 legacy=0x7efe6c0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 10 monclient: shutdown 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 >> v1:172.21.15.133:6789/0 conn(0x7efe8c13f710 legacy=0x7efe8c1d4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 shutdown_connections 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.748+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 >> 172.21.15.55:0/3731353203 conn(0x7efe8c0b7ea0 msgr2=0x7efe8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:24.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.749+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 shutdown_connections 2024-03-25T18:32:24.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.749+0000 7efe9160d640 1 -- 172.21.15.55:0/3731353203 wait complete. 2024-03-25T18:32:24.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:24.749+0000 7efe9160d640 1 librados: shutdown 2024-03-25T18:32:24.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:31.567 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.677+0000 7fb1ec8f6640 10 monclient: get_monmap_and_config 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.677+0000 7fb1ec8f6640 10 monclient: build_initial_monmap 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.677+0000 7fb1ec8f6640 10 monclient: monmap: 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:32:31.678215+0000 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:32:31.678215+0000 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:31.678 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:32:31.679 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:32:31.679 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.677+0000 7fb1ec8f6640 1 Processor -- start 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.677+0000 7fb1ec8f6640 1 -- start start 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.677+0000 7fb1ec8f6640 10 monclient: init 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e80b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e80b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e814be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80af940 con 0x7fb1e80b4d60 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80b0a70 con 0x7fb1e814be90 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80b0d50 con 0x7fb1e80b1730 2024-03-25T18:32:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:31.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.678+0000 7fb1ec8f6640 10 monclient(hunting): authenticate will time out at 7555.936115s 2024-03-25T18:32:31.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.679+0000 7fb1e5d9b640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fb1e80b1730 0x7fb1e80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:51202/0 (socket says 172.21.15.55:51202) 2024-03-25T18:32:31.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.679+0000 7fb1e5d9b640 1 -- 172.21.15.55:0/709178515 learned_addr learned my addr 172.21.15.55:0/709178515 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:31.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.707+0000 7fb1e559a640 1 -- 172.21.15.55:0/709178515 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227647104 0 0) 0x7fb1e80af940 con 0x7fb1e80b4d60 2024-03-25T18:32:31.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.707+0000 7fb1e559a640 10 monclient(hunting): _negotiate 2024-03-25T18:32:31.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.707+0000 7fb1e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:31.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.707+0000 7fb1e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:31.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.707+0000 7fb1e559a640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:32:31.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.707+0000 7fb1e559a640 1 -- 172.21.15.55:0/709178515 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d4003520 con 0x7fb1e80b4d60 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 1 -- 172.21.15.55:0/709178515 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1301881983 0 0) 0x7fb1d4003520 con 0x7fb1e80b4d60 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 1 -- 172.21.15.55:0/709178515 >> v1:172.21.15.55:6790/0 conn(0x7fb1e80b1730 legacy=0x7fb1e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 1 -- 172.21.15.55:0/709178515 >> v1:172.21.15.133:6789/0 conn(0x7fb1e814be90 legacy=0x7fb1e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 10 monclient: _finish_hunting 0 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 1 monclient: found mon.noname-a 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 1 -- 172.21.15.55:0/709178515 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e80b1030 con 0x7fb1e80b4d60 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 10 monclient: _finish_auth 0 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 10 monclient: _check_auth_tickets 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 1 -- 172.21.15.55:0/709178515 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb1d8001a10 con 0x7fb1e80b4d60 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1ec8f6640 5 monclient: authenticate success, global_id 4428 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1ec8f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1e559a640 10 monclient: dump: 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.714+0000 7fb1ec8f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:31.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.727+0000 7fb1e559a640 1 -- 172.21.15.55:0/709178515 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1d8001be0 con 0x7fb1e80b4d60 2024-03-25T18:32:31.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.727+0000 7fb1e559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:31.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.727+0000 7fb1ec8f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:31.728 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.727+0000 7fb1ec8f6640 4 set_mon_vals no callback set 2024-03-25T18:32:31.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.728+0000 7fb1ec8f6640 10 monclient: shutdown 2024-03-25T18:32:31.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.728+0000 7fb1ec8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:31.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.728+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/709178515 >> v1:172.21.15.55:6789/0 conn(0x7fb1e80b4d60 legacy=0x7fb1e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.728+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/709178515 shutdown_connections 2024-03-25T18:32:31.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.728+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/709178515 >> 172.21.15.55:0/709178515 conn(0x7fb1e80617a0 msgr2=0x7fb1e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:31.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.728+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/709178515 shutdown_connections 2024-03-25T18:32:31.729 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.728+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/709178515 wait complete. 2024-03-25T18:32:31.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.729+0000 7fb1ec8f6640 10 monclient: build_initial_monmap 2024-03-25T18:32:31.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.729+0000 7fb1ec8f6640 10 monclient: monmap: 2024-03-25T18:32:31.731 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:31.731 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:31.731 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:32:31.731 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:32:31.731 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:31.731 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.729+0000 7fb1ec8f6640 1 librados: starting msgr at 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.729+0000 7fb1ec8f6640 1 librados: starting objecter 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.729+0000 7fb1ec8f6640 1 Processor -- start 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.729+0000 7fb1ec8f6640 1 -- start start 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.729+0000 7fb1ec8f6640 1 librados: setting wanted keys 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.729+0000 7fb1ec8f6640 1 librados: calling monclient init 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.729+0000 7fb1ec8f6640 10 monclient: init 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e80b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e80b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:31.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e814be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:31.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e81d7620 con 0x7fb1e80b4d60 2024-03-25T18:32:31.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80aa8d0 con 0x7fb1e80b1730 2024-03-25T18:32:31.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80aab40 con 0x7fb1e814be90 2024-03-25T18:32:31.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:31.733 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.730+0000 7fb1ec8f6640 10 monclient(hunting): authenticate will time out at 7555.987480s 2024-03-25T18:32:31.740 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.740+0000 7fb1e659c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fb1e80b1730 0x7fb1e8137e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:52512/0 (socket says 172.21.15.55:52512) 2024-03-25T18:32:31.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.740+0000 7fb1e659c640 1 -- 172.21.15.55:0/2262490511 learned_addr learned my addr 172.21.15.55:0/2262490511 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:31.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.740+0000 7fb1deffd640 10 client.?.objecter ms_handle_connect 0x7fb1e80b1730 2024-03-25T18:32:31.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.740+0000 7fb1deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:31.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.749+0000 7fb1deffd640 10 client.?.objecter ms_handle_connect 0x7fb1e80b4d60 2024-03-25T18:32:31.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.749+0000 7fb1deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:31.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.772+0000 7fb1deffd640 10 client.?.objecter ms_handle_connect 0x7fb1e814be90 2024-03-25T18:32:31.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.772+0000 7fb1deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.773+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776225940 0 0) 0x7fb1e81d7620 con 0x7fb1e80b4d60 2024-03-25T18:32:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.773+0000 7fb1deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.773+0000 7fb1deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.773+0000 7fb1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.773+0000 7fb1deffd640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:32:31.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.773+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c8004eb0 con 0x7fb1e80b4d60 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.798+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2590428486 0 0) 0x7fb1c8004eb0 con 0x7fb1e80b4d60 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.798+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e81d7620 con 0x7fb1e80b4d60 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.798+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb1d00018a0 con 0x7fb1e80b4d60 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.798+0000 7fb1deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.798+0000 7fb1deffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.798+0000 7fb1deffd640 10 monclient(hunting): dump: 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:31.799 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:31.800 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:31.800 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:31.800 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:31.800 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:31.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.798+0000 7fb1deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:31.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2086129228 0 0) 0x7fb1e81d7620 con 0x7fb1e80b4d60 2024-03-25T18:32:31.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 >> v1:172.21.15.55:6790/0 conn(0x7fb1e814be90 legacy=0x7fb1e81e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 >> v1:172.21.15.133:6789/0 conn(0x7fb1e80b1730 legacy=0x7fb1e8137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 1 monclient: found mon.a 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8138dd0 con 0x7fb1e80b4d60 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1ec8f6640 5 monclient: authenticate success, global_id 4431 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1ec8f6640 10 monclient: _renew_subs 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1ec8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e813b610 con 0x7fb1e80b4d60 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1ec8f6640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1ec8f6640 10 monclient: _renew_subs 2024-03-25T18:32:31.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1ec8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:31.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e813b3c0 con 0x7fb1e80b4d60 2024-03-25T18:32:31.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.859+0000 7fb1ec8f6640 1 librados: init done 2024-03-25T18:32:31.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.860+0000 7fb1ec8f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:31.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.860+0000 7fb1ec8f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:31.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.860+0000 7fb1ec8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:31.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:31.860+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e81d77d0 con 0x7fb1e80b4d60 2024-03-25T18:32:32.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.177+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1d0001d50 con 0x7fb1e80b4d60 2024-03-25T18:32:32.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.177+0000 7fb1deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:32.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.177+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb1d0004860 con 0x7fb1e80b4d60 2024-03-25T18:32:32.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.177+0000 7fb1deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.177+0000 7fb1deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.177+0000 7fb1deffd640 10 monclient: dump: 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:32.179 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:32.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.177+0000 7fb1dffff640 4 set_mon_vals no callback set 2024-03-25T18:32:32.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.178+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fb1d0004a80 con 0x7fb1e80b4d60 2024-03-25T18:32:32.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.178+0000 7fb1e659c640 10 monclient: get_auth_request con 0x7fb1c806dc00 auth_method 0 2024-03-25T18:32:32.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.179+0000 7fb1deffd640 10 client.4431.objecter ms_handle_connect 0x7fb1c806dc00 2024-03-25T18:32:32.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.184+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fb1d0054660 con 0x7fb1e80b4d60 2024-03-25T18:32:32.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.184+0000 7fb1deffd640 10 client.4431.objecter ms_dispatch 0x7fb1e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:32.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.184+0000 7fb1deffd640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:32.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.184+0000 7fb1deffd640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:32.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.184+0000 7fb1deffd640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-25T18:32:32.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.184+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1d004ffa0 con 0x7fb1e80b4d60 2024-03-25T18:32:32.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.184+0000 7fb1deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:32.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.184+0000 7fb1deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:32.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.456+0000 7fb1ec8f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:32.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.456+0000 7fb1ec8f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:32.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.456+0000 7fb1ec8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:32.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.456+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e80627b0 con 0x7fb1e80b4d60 2024-03-25T18:32:32.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.492+0000 7fb1deffd640 1 -- 172.21.15.55:0/2262490511 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fb1d00520a0 con 0x7fb1e80b4d60 2024-03-25T18:32:32.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.492+0000 7fb1deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:32.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.492+0000 7fb1deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:32.493 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:32:32.493 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:32.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 10 librados: watch_flush enter 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 10 librados: watch_flush exit 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 10 client.4431.objecter successfully canceled tick 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 >> v1:172.21.15.55:6800/286536064 conn(0x7fb1c806dc00 legacy=0x7fb1c80700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 10 monclient: shutdown 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 >> v1:172.21.15.55:6789/0 conn(0x7fb1e80b4d60 legacy=0x7fb1e81d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 shutdown_connections 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 >> 172.21.15.55:0/2262490511 conn(0x7fb1e80617a0 msgr2=0x7fb1e8145170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 shutdown_connections 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 1 -- 172.21.15.55:0/2262490511 wait complete. 2024-03-25T18:32:32.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:32.494+0000 7fb1ec8f6640 1 librados: shutdown 2024-03-25T18:32:32.508 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:35.088 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:04:45.258 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:35.096 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:04:45.266 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:35.132 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:04:45.302 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:35.167 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:04:45.337 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:35.170 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:04:45.341 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:35.181 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:04:45.319 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:35.187 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:04:45.324 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:35.194 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:04:45.364 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:35.210 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:04:45.347 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:39.314 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:39.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.428+0000 7f27df3b9640 10 monclient: get_monmap_and_config 2024-03-25T18:32:39.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.428+0000 7f27df3b9640 10 monclient: build_initial_monmap 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.428+0000 7f27df3b9640 10 monclient: monmap: 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:32:39.429754+0000 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:32:39.429754+0000 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:32:39.430 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:39.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.429+0000 7f27df3b9640 1 Processor -- start 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.429+0000 7f27df3b9640 1 -- start start 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.429+0000 7f27df3b9640 10 monclient: init 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 10 monclient(hunting): picked mon.noname-a con 0x7f27d80b4f10 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 10 monclient(hunting): picked mon.noname-c con 0x7f27d80b44d0 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 10 monclient(hunting): picked mon.noname-b con 0x7f27d813f740 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d80b0140 con 0x7f27d80b4f10 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d80af3a0 con 0x7f27d80b44d0 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d80af530 con 0x7f27d813f740 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27df3b9640 10 monclient(hunting): authenticate will time out at 7563.687629s 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27dd956640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f27d813f740 0x7f27d813fbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:50488/0 (socket says 172.21.15.55:50488) 2024-03-25T18:32:39.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.430+0000 7f27dd956640 1 -- 172.21.15.55:0/3373783760 learned_addr learned my addr 172.21.15.55:0/3373783760 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:39.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.491+0000 7f27cffff640 1 -- 172.21.15.55:0/3373783760 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369088959 0 0) 0x7f27d80b0140 con 0x7f27d80b4f10 2024-03-25T18:32:39.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.491+0000 7f27cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:39.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.491+0000 7f27cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:39.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.491+0000 7f27cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:39.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.491+0000 7f27cffff640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:32:39.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.491+0000 7f27cffff640 1 -- 172.21.15.55:0/3373783760 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27c4003520 con 0x7f27d80b4f10 2024-03-25T18:32:39.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 1 -- 172.21.15.55:0/3373783760 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2446663180 0 0) 0x7f27c4003520 con 0x7f27d80b4f10 2024-03-25T18:32:39.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 1 -- 172.21.15.55:0/3373783760 >> v1:172.21.15.55:6790/0 conn(0x7f27d813f740 legacy=0x7f27d813fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:39.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 1 -- 172.21.15.55:0/3373783760 >> v1:172.21.15.133:6789/0 conn(0x7f27d80b44d0 legacy=0x7f27d80b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:39.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:39.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 1 monclient: found mon.noname-a 2024-03-25T18:32:39.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 1 -- 172.21.15.55:0/3373783760 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d80afcd0 con 0x7f27d80b4f10 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 10 monclient: _finish_auth 0 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 1 -- 172.21.15.55:0/3373783760 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f27c8001690 con 0x7f27d80b4f10 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27cffff640 10 monclient: dump: 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.510 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:39.511 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:39.511 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:39.511 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:39.511 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:39.511 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:39.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27df3b9640 5 monclient: authenticate success, global_id 4434 2024-03-25T18:32:39.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.508+0000 7f27df3b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:39.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27cffff640 1 -- 172.21.15.55:0/3373783760 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27c80017f0 con 0x7f27d80b4f10 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27cffff640 1 -- 172.21.15.55:0/3373783760 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f27c8001c40 con 0x7f27d80b4f10 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27df3b9640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27df3b9640 4 set_mon_vals no callback set 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27cffff640 10 monclient: dump: 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:39.513 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:39.514 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:39.514 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27df3b9640 10 monclient: shutdown 2024-03-25T18:32:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27df3b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27df3b9640 1 -- 172.21.15.55:0/3373783760 >> v1:172.21.15.55:6789/0 conn(0x7f27d80b4f10 legacy=0x7f27d80ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27df3b9640 1 -- 172.21.15.55:0/3373783760 shutdown_connections 2024-03-25T18:32:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.512+0000 7f27df3b9640 1 -- 172.21.15.55:0/3373783760 >> 172.21.15.55:0/3373783760 conn(0x7f27d80617a0 msgr2=0x7f27d8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:39.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.513+0000 7f27df3b9640 1 -- 172.21.15.55:0/3373783760 shutdown_connections 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.513+0000 7f27df3b9640 1 -- 172.21.15.55:0/3373783760 wait complete. 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.514+0000 7f27df3b9640 10 monclient: build_initial_monmap 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.514+0000 7f27df3b9640 10 monclient: monmap: 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.514+0000 7f27df3b9640 1 librados: starting msgr at 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.514+0000 7f27df3b9640 1 librados: starting objecter 2024-03-25T18:32:39.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.514+0000 7f27df3b9640 1 Processor -- start 2024-03-25T18:32:39.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.514+0000 7f27df3b9640 1 -- start start 2024-03-25T18:32:39.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.514+0000 7f27df3b9640 1 librados: setting wanted keys 2024-03-25T18:32:39.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.514+0000 7f27df3b9640 1 librados: calling monclient init 2024-03-25T18:32:39.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.514+0000 7f27df3b9640 10 monclient: init 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 10 monclient(hunting): picked mon.noname-c con 0x7f27d80b44d0 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 10 monclient(hunting): picked mon.noname-b con 0x7f27d80b4f10 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 10 monclient(hunting): picked mon.noname-a con 0x7f27d813f740 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d813af40 con 0x7f27d813f740 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d81467f0 con 0x7f27d80b4f10 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d8146530 con 0x7f27d80b44d0 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:39.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.515+0000 7f27df3b9640 10 monclient(hunting): authenticate will time out at 7563.773221s 2024-03-25T18:32:39.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.520+0000 7f27dd956640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f27d813f740 0x7f27d81db920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:45718/0 (socket says 172.21.15.55:45718) 2024-03-25T18:32:39.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.520+0000 7f27dd956640 1 -- 172.21.15.55:0/4227857071 learned_addr learned my addr 172.21.15.55:0/4227857071 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:39.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.522+0000 7f27cdffb640 10 client.?.objecter ms_handle_connect 0x7f27d813f740 2024-03-25T18:32:39.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.522+0000 7f27cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:39.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.534+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354417786 0 0) 0x7f27d813af40 con 0x7f27d813f740 2024-03-25T18:32:39.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.534+0000 7f27cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:39.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.534+0000 7f27cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:39.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.534+0000 7f27cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:39.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.534+0000 7f27cdffb640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:32:39.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.534+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27bc004eb0 con 0x7f27d813f740 2024-03-25T18:32:39.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.547+0000 7f27cdffb640 10 client.?.objecter ms_handle_connect 0x7f27d80b4f10 2024-03-25T18:32:39.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.547+0000 7f27cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:39.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.549+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1317314411 0 0) 0x7f27bc004eb0 con 0x7f27d813f740 2024-03-25T18:32:39.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.549+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27d813af40 con 0x7f27d813f740 2024-03-25T18:32:39.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.549+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f27d0001b10 con 0x7f27d813f740 2024-03-25T18:32:39.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.549+0000 7f27cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:39.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.549+0000 7f27cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:39.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.549+0000 7f27cdffb640 10 monclient(hunting): dump: 2024-03-25T18:32:39.550 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:39.550 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:39.551 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.551 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.551 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:39.551 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:39.551 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:39.551 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:39.551 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:39.551 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:39.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.549+0000 7f27cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:39.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.555+0000 7f27cdffb640 10 client.?.objecter ms_handle_connect 0x7f27d80b44d0 2024-03-25T18:32:39.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.555+0000 7f27cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:39.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.559+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178513632 0 0) 0x7f27d81467f0 con 0x7f27d80b4f10 2024-03-25T18:32:39.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.559+0000 7f27cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:39.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.559+0000 7f27cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:39.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.559+0000 7f27cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:39.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.559+0000 7f27cdffb640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:32:39.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.559+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27bc004eb0 con 0x7f27d80b4f10 2024-03-25T18:32:39.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.565+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519025736 0 0) 0x7f27d8146530 con 0x7f27d80b44d0 2024-03-25T18:32:39.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.565+0000 7f27cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:39.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.565+0000 7f27cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:39.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.565+0000 7f27cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:39.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.565+0000 7f27cdffb640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:32:39.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.566+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d81467f0 con 0x7f27d80b44d0 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.570+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1261766685 0 0) 0x7f27bc004eb0 con 0x7f27d80b4f10 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.570+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27d8146530 con 0x7f27d80b4f10 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.570+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f27d4001860 con 0x7f27d80b4f10 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.570+0000 7f27cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.570+0000 7f27cdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.570+0000 7f27cdffb640 10 monclient(hunting): dump: 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:39.571 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:39.572 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:39.572 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:39.572 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:39.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.578+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2481860090 0 0) 0x7f27d81467f0 con 0x7f27d80b44d0 2024-03-25T18:32:39.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.578+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27bc004eb0 con 0x7f27d80b44d0 2024-03-25T18:32:39.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.578+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f27c8001b40 con 0x7f27d80b44d0 2024-03-25T18:32:39.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.578+0000 7f27cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:39.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.578+0000 7f27cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:39.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.578+0000 7f27cdffb640 10 monclient(hunting): dump: 2024-03-25T18:32:39.579 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:39.580 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:39.580 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.580 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.580 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:39.580 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:39.580 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:39.580 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:39.580 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:39.580 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:39.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.586+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2095835186 0 0) 0x7f27d813af40 con 0x7f27d813f740 2024-03-25T18:32:39.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.586+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 >> v1:172.21.15.55:6790/0 conn(0x7f27d80b44d0 legacy=0x7f27d81db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:39.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.586+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 >> v1:172.21.15.133:6789/0 conn(0x7f27d80b4f10 legacy=0x7f27d81424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:39.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.586+0000 7f27cdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:39.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.586+0000 7f27cdffb640 1 monclient: found mon.a 2024-03-25T18:32:39.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.586+0000 7f27cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:39.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:39.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d8136010 con 0x7f27d813f740 2024-03-25T18:32:39.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27cdffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27cdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27df3b9640 5 monclient: authenticate success, global_id 4437 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27df3b9640 10 monclient: _renew_subs 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27df3b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d81458e0 con 0x7f27d813f740 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27df3b9640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27df3b9640 10 monclient: _renew_subs 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27df3b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d8145e30 con 0x7f27d813f740 2024-03-25T18:32:39.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.587+0000 7f27df3b9640 1 librados: init done 2024-03-25T18:32:39.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.588+0000 7f27df3b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:39.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.588+0000 7f27df3b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:39.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.588+0000 7f27df3b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:39.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.588+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d80627b0 con 0x7f27d813f740 2024-03-25T18:32:39.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.903+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27d0002080 con 0x7f27d813f740 2024-03-25T18:32:39.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.903+0000 7f27cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.903+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f27d0004d00 con 0x7f27d813f740 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.903+0000 7f27cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.903+0000 7f27cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.903+0000 7f27cdffb640 10 monclient: dump: 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:39.905 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:39.906 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:39.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.903+0000 7f27cf7fe640 4 set_mon_vals no callback set 2024-03-25T18:32:39.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.904+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f27d0004f20 con 0x7f27d813f740 2024-03-25T18:32:39.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.905+0000 7f27dd155640 10 monclient: get_auth_request con 0x7f27bc06e180 auth_method 0 2024-03-25T18:32:39.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.905+0000 7f27cdffb640 10 client.4437.objecter ms_handle_connect 0x7f27bc06e180 2024-03-25T18:32:39.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.912+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f27d0054a00 con 0x7f27d813f740 2024-03-25T18:32:39.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.912+0000 7f27cdffb640 10 client.4437.objecter ms_dispatch 0x7f27d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:39.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.912+0000 7f27cdffb640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:39.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.912+0000 7f27cdffb640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:39.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.912+0000 7f27cdffb640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T18:32:39.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.912+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27d0050510 con 0x7f27d813f740 2024-03-25T18:32:39.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.912+0000 7f27cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:39.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:39.912+0000 7f27cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:40.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.172+0000 7f27df3b9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:40.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.172+0000 7f27df3b9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:40.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.172+0000 7f27df3b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:40.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.172+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27d8143ab0 con 0x7f27d813f740 2024-03-25T18:32:40.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.194+0000 7f27cdffb640 1 -- 172.21.15.55:0/4227857071 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f27d0052440 con 0x7f27d813f740 2024-03-25T18:32:40.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.194+0000 7f27cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:40.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.194+0000 7f27cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:40.196 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:32:40.196 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:40.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 10 librados: watch_flush enter 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 10 librados: watch_flush exit 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 10 client.4437.objecter successfully canceled tick 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 >> v1:172.21.15.55:6800/286536064 conn(0x7f27bc06e180 legacy=0x7f27bc070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 10 monclient: shutdown 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 >> v1:172.21.15.55:6789/0 conn(0x7f27d813f740 legacy=0x7f27d81db920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 shutdown_connections 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 >> 172.21.15.55:0/4227857071 conn(0x7f27d80617a0 msgr2=0x7f27d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 shutdown_connections 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 1 -- 172.21.15.55:0/4227857071 wait complete. 2024-03-25T18:32:40.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:40.197+0000 7f27df3b9640 1 librados: shutdown 2024-03-25T18:32:40.209 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:47.017 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:47.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.130+0000 7f2e8556b640 10 monclient: get_monmap_and_config 2024-03-25T18:32:47.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.130+0000 7f2e8556b640 10 monclient: build_initial_monmap 2024-03-25T18:32:47.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.131+0000 7f2e8556b640 10 monclient: monmap: 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:32:47.132858+0000 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:32:47.132858+0000 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.131+0000 7f2e8556b640 1 Processor -- start 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.131+0000 7f2e8556b640 1 -- start start 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.131+0000 7f2e8556b640 10 monclient: init 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:47.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 10 monclient(hunting): picked mon.noname-c con 0x7f2e80061830 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:47.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e8006e740 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:47.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e800ae280 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:47.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e800ae830 con 0x7f2e800ae280 2024-03-25T18:32:47.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8006f720 con 0x7f2e80061830 2024-03-25T18:32:47.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e800ada80 con 0x7f2e8006e740 2024-03-25T18:32:47.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:47.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.132+0000 7f2e8556b640 10 monclient(hunting): authenticate will time out at 7571.391177s 2024-03-25T18:32:47.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.133+0000 7f2e7effd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2e80061830 0x7f2e8013f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:38748/0 (socket says 172.21.15.55:38748) 2024-03-25T18:32:47.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.133+0000 7f2e7effd640 1 -- 172.21.15.55:0/2038348807 learned_addr learned my addr 172.21.15.55:0/2038348807 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:47.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.216+0000 7f2e7dffb640 1 -- 172.21.15.55:0/2038348807 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682712483 0 0) 0x7f2e800ae830 con 0x7f2e800ae280 2024-03-25T18:32:47.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.216+0000 7f2e7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:47.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.216+0000 7f2e7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:47.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.216+0000 7f2e7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:47.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.216+0000 7f2e7dffb640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:32:47.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.216+0000 7f2e7dffb640 1 -- 172.21.15.55:0/2038348807 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e68003520 con 0x7f2e800ae280 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 1 -- 172.21.15.55:0/2038348807 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517881445 0 0) 0x7f2e68003520 con 0x7f2e800ae280 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 1 -- 172.21.15.55:0/2038348807 >> v1:172.21.15.55:6790/0 conn(0x7f2e8006e740 legacy=0x7f2e80145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 1 -- 172.21.15.55:0/2038348807 >> v1:172.21.15.133:6789/0 conn(0x7f2e80061830 legacy=0x7f2e8013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 1 monclient: found mon.noname-a 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 1 -- 172.21.15.55:0/2038348807 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8013fcd0 con 0x7f2e800ae280 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:47.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 1 -- 172.21.15.55:0/2038348807 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2e78001a10 con 0x7f2e800ae280 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e7dffb640 10 monclient: dump: 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e8556b640 5 monclient: authenticate success, global_id 4440 2024-03-25T18:32:47.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.224+0000 7f2e8556b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e7dffb640 1 -- 172.21.15.55:0/2038348807 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e78001b70 con 0x7f2e800ae280 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e7dffb640 1 -- 172.21.15.55:0/2038348807 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2e78001f80 con 0x7f2e800ae280 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e8556b640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e8556b640 4 set_mon_vals no callback set 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e7dffb640 10 monclient: dump: 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:47.235 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:47.236 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:47.236 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:47.236 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:47.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e8556b640 10 monclient: shutdown 2024-03-25T18:32:47.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e8556b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:47.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e8556b640 1 -- 172.21.15.55:0/2038348807 >> v1:172.21.15.55:6789/0 conn(0x7f2e800ae280 legacy=0x7f2e80145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:47.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e8556b640 1 -- 172.21.15.55:0/2038348807 shutdown_connections 2024-03-25T18:32:47.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.233+0000 7f2e8556b640 1 -- 172.21.15.55:0/2038348807 >> 172.21.15.55:0/2038348807 conn(0x7f2e800b0ef0 msgr2=0x7f2e800b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:47.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.234+0000 7f2e8556b640 1 -- 172.21.15.55:0/2038348807 shutdown_connections 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.234+0000 7f2e8556b640 1 -- 172.21.15.55:0/2038348807 wait complete. 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.234+0000 7f2e8556b640 10 monclient: build_initial_monmap 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.234+0000 7f2e8556b640 10 monclient: monmap: 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:32:47.238 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:47.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.235+0000 7f2e8556b640 1 librados: starting msgr at 2024-03-25T18:32:47.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.235+0000 7f2e8556b640 1 librados: starting objecter 2024-03-25T18:32:47.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.235+0000 7f2e8556b640 1 Processor -- start 2024-03-25T18:32:47.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.235+0000 7f2e8556b640 1 -- start start 2024-03-25T18:32:47.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.235+0000 7f2e8556b640 1 librados: setting wanted keys 2024-03-25T18:32:47.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.235+0000 7f2e8556b640 1 librados: calling monclient init 2024-03-25T18:32:47.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.235+0000 7f2e8556b640 10 monclient: init 2024-03-25T18:32:47.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:47.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:47.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 10 monclient(hunting): picked mon.noname-a con 0x7f2e80061830 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:47.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 10 monclient(hunting): picked mon.noname-b con 0x7f2e800ae280 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:47.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 10 monclient(hunting): picked mon.noname-c con 0x7f2e801462f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:47.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e80145be0 con 0x7f2e80061830 2024-03-25T18:32:47.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e801d7a20 con 0x7f2e800ae280 2024-03-25T18:32:47.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e80144660 con 0x7f2e801462f0 2024-03-25T18:32:47.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:47.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.236+0000 7f2e8556b640 10 monclient(hunting): authenticate will time out at 7571.494927s 2024-03-25T18:32:47.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.252+0000 7f2e7effd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2e80061830 0x7f2e801d7310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:50422/0 (socket says 172.21.15.55:50422) 2024-03-25T18:32:47.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.252+0000 7f2e7effd640 1 -- 172.21.15.55:0/1080277435 learned_addr learned my addr 172.21.15.55:0/1080277435 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:47.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.253+0000 7f2e677fe640 10 client.?.objecter ms_handle_connect 0x7f2e80061830 2024-03-25T18:32:47.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.253+0000 7f2e677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:47.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.265+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194134149 0 0) 0x7f2e80145be0 con 0x7f2e80061830 2024-03-25T18:32:47.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.265+0000 7f2e677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:47.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.265+0000 7f2e677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:47.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.265+0000 7f2e677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:47.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.265+0000 7f2e677fe640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:32:47.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.265+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e6c004eb0 con 0x7f2e80061830 2024-03-25T18:32:47.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.270+0000 7f2e677fe640 10 client.?.objecter ms_handle_connect 0x7f2e800ae280 2024-03-25T18:32:47.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.270+0000 7f2e677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:47.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.282+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259735651 0 0) 0x7f2e801d7a20 con 0x7f2e800ae280 2024-03-25T18:32:47.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.282+0000 7f2e677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:47.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.282+0000 7f2e677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:47.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.282+0000 7f2e677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:47.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.282+0000 7f2e677fe640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:32:47.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.282+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e80145be0 con 0x7f2e800ae280 2024-03-25T18:32:47.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.282+0000 7f2e677fe640 10 client.?.objecter ms_handle_connect 0x7f2e801462f0 2024-03-25T18:32:47.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.282+0000 7f2e677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:47.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.283+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3382648831 0 0) 0x7f2e6c004eb0 con 0x7f2e80061830 2024-03-25T18:32:47.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.283+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e801d7a20 con 0x7f2e80061830 2024-03-25T18:32:47.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.283+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2e700018b0 con 0x7f2e80061830 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.283+0000 7f2e677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.283+0000 7f2e677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.284+0000 7f2e677fe640 10 monclient(hunting): dump: 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:47.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.284+0000 7f2e677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:47.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.304+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2110578829 0 0) 0x7f2e80145be0 con 0x7f2e800ae280 2024-03-25T18:32:47.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.304+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e6c004eb0 con 0x7f2e800ae280 2024-03-25T18:32:47.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.304+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2e740018a0 con 0x7f2e800ae280 2024-03-25T18:32:47.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.304+0000 7f2e677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:47.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.304+0000 7f2e677fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:47.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.304+0000 7f2e677fe640 10 monclient(hunting): dump: 2024-03-25T18:32:47.306 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:47.306 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.305+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786427735 0 0) 0x7f2e80144660 con 0x7f2e801462f0 2024-03-25T18:32:47.307 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.305+0000 7f2e677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:47.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.305+0000 7f2e677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:47.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.305+0000 7f2e677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:47.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.305+0000 7f2e677fe640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:32:47.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.305+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e80145be0 con 0x7f2e801462f0 2024-03-25T18:32:47.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3726645226 0 0) 0x7f2e801d7a20 con 0x7f2e80061830 2024-03-25T18:32:47.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 >> v1:172.21.15.55:6790/0 conn(0x7f2e801462f0 legacy=0x7f2e80143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:47.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 >> v1:172.21.15.133:6789/0 conn(0x7f2e800ae280 legacy=0x7f2e8013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:47.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 1 monclient: found mon.a 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8013f6d0 con 0x7f2e80061830 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e8556b640 5 monclient: authenticate success, global_id 4443 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e8556b640 10 monclient: _renew_subs 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e8556b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e80143d30 con 0x7f2e80061830 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e8556b640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e8556b640 10 monclient: _renew_subs 2024-03-25T18:32:47.316 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e8556b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:47.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e801442f0 con 0x7f2e80061830 2024-03-25T18:32:47.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.313+0000 7f2e8556b640 1 librados: init done 2024-03-25T18:32:47.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.314+0000 7f2e8556b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:47.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.314+0000 7f2e8556b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:47.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.314+0000 7f2e8556b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:47.317 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.314+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e80145f90 con 0x7f2e80061830 2024-03-25T18:32:47.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.612+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e700047e0 con 0x7f2e80061830 2024-03-25T18:32:47.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.612+0000 7f2e677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.612+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2e70004c30 con 0x7f2e80061830 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.612+0000 7f2e677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.612+0000 7f2e677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.612+0000 7f2e677fe640 10 monclient: dump: 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:47.615 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:47.616 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:47.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.612+0000 7f2e7d7fa640 4 set_mon_vals no callback set 2024-03-25T18:32:47.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.613+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f2e70004e50 con 0x7f2e80061830 2024-03-25T18:32:47.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.614+0000 7f2e7e7fc640 10 monclient: get_auth_request con 0x7f2e6c06df40 auth_method 0 2024-03-25T18:32:47.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.614+0000 7f2e677fe640 10 client.4443.objecter ms_handle_connect 0x7f2e6c06df40 2024-03-25T18:32:47.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.622+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f2e700547c0 con 0x7f2e80061830 2024-03-25T18:32:47.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.622+0000 7f2e677fe640 10 client.4443.objecter ms_dispatch 0x7f2e80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:47.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.622+0000 7f2e677fe640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:47.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.622+0000 7f2e677fe640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:47.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.622+0000 7f2e677fe640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T18:32:47.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.622+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e700502d0 con 0x7f2e80061830 2024-03-25T18:32:47.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.622+0000 7f2e677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:47.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.622+0000 7f2e677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:47.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.884+0000 7f2e8556b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:47.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.884+0000 7f2e8556b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:47.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.884+0000 7f2e8556b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:47.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.884+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e8013f860 con 0x7f2e80061830 2024-03-25T18:32:47.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.915+0000 7f2e677fe640 1 -- 172.21.15.55:0/1080277435 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f2e70052200 con 0x7f2e80061830 2024-03-25T18:32:47.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.915+0000 7f2e677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:47.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.915+0000 7f2e677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:47.917 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:32:47.917 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.917+0000 7f2e8556b640 10 librados: watch_flush enter 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.917+0000 7f2e8556b640 10 librados: watch_flush exit 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 10 client.4443.objecter successfully canceled tick 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 >> v1:172.21.15.55:6800/286536064 conn(0x7f2e6c06df40 legacy=0x7f2e6c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 10 monclient: shutdown 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 >> v1:172.21.15.55:6789/0 conn(0x7f2e80061830 legacy=0x7f2e801d7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 shutdown_connections 2024-03-25T18:32:47.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 >> 172.21.15.55:0/1080277435 conn(0x7f2e800b0ef0 msgr2=0x7f2e800711f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:47.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 shutdown_connections 2024-03-25T18:32:47.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 1 -- 172.21.15.55:0/1080277435 wait complete. 2024-03-25T18:32:47.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:47.918+0000 7f2e8556b640 1 librados: shutdown 2024-03-25T18:32:47.933 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:54.739 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.851+0000 7ff6c6cd6640 10 monclient: get_monmap_and_config 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.851+0000 7ff6c6cd6640 10 monclient: build_initial_monmap 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.851+0000 7ff6c6cd6640 10 monclient: monmap: 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:32:54.851896+0000 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:32:54.851896+0000 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:32:54.852 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:32:54.853 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:54.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.851+0000 7ff6c6cd6640 1 Processor -- start 2024-03-25T18:32:54.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.851+0000 7ff6c6cd6640 1 -- start start 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 10 monclient: init 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c0061830 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c013c8c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c0072aa0 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c0071d40 con 0x7ff6c013c8c0 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c0072ef0 con 0x7ff6c0061830 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c0071f70 con 0x7ff6c0072aa0 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:54.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.852+0000 7ff6c6cd6640 10 monclient(hunting): authenticate will time out at 7579.110136s 2024-03-25T18:32:54.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.861+0000 7ff6c5273640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7ff6c0072aa0 0x7ff6c014a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:44282/0 (socket says 172.21.15.55:44282) 2024-03-25T18:32:54.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.861+0000 7ff6c5273640 1 -- 172.21.15.55:0/362392565 learned_addr learned my addr 172.21.15.55:0/362392565 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:54.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.880+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169535752 0 0) 0x7ff6c0071f70 con 0x7ff6c0072aa0 2024-03-25T18:32:54.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.880+0000 7ff6b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:54.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.880+0000 7ff6b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:54.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.880+0000 7ff6b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:54.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.880+0000 7ff6b77fe640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:32:54.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.880+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a8003520 con 0x7ff6c0072aa0 2024-03-25T18:32:54.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.881+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766855484 0 0) 0x7ff6c0071d40 con 0x7ff6c013c8c0 2024-03-25T18:32:54.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.881+0000 7ff6b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:54.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.881+0000 7ff6b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:54.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.881+0000 7ff6b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:54.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.881+0000 7ff6b77fe640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:32:54.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.881+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c0071f70 con 0x7ff6c013c8c0 2024-03-25T18:32:54.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.885+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275665557 0 0) 0x7ff6c0072ef0 con 0x7ff6c0061830 2024-03-25T18:32:54.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.885+0000 7ff6b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:54.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.885+0000 7ff6b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:54.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.885+0000 7ff6b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:54.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.885+0000 7ff6b77fe640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:32:54.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.885+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c0071d40 con 0x7ff6c0061830 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816885588 0 0) 0x7ff6c0071f70 con 0x7ff6c013c8c0 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 >> v1:172.21.15.55:6790/0 conn(0x7ff6c0072aa0 legacy=0x7ff6c014a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 >> v1:172.21.15.133:6789/0 conn(0x7ff6c0061830 legacy=0x7ff6c013c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 1 monclient: found mon.noname-a 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c006fbb0 con 0x7ff6c013c8c0 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:54.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff6b0001670 con 0x7ff6c013c8c0 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6c6cd6640 5 monclient: authenticate success, global_id 4446 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6c6cd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6b77fe640 10 monclient: dump: 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:54.890 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:54.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.887+0000 7ff6c6cd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:54.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.899+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6b00017d0 con 0x7ff6c013c8c0 2024-03-25T18:32:54.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.899+0000 7ff6b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:54.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.899+0000 7ff6b77fe640 1 -- 172.21.15.55:0/362392565 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff6b0001be0 con 0x7ff6c013c8c0 2024-03-25T18:32:54.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.899+0000 7ff6c6cd6640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:54.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.899+0000 7ff6c6cd6640 4 set_mon_vals no callback set 2024-03-25T18:32:54.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.899+0000 7ff6b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:54.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.899+0000 7ff6b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:54.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.899+0000 7ff6b77fe640 10 monclient: dump: 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:54.901 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:54.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.900+0000 7ff6c6cd6640 10 monclient: shutdown 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.900+0000 7ff6c6cd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.900+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/362392565 >> v1:172.21.15.55:6789/0 conn(0x7ff6c013c8c0 legacy=0x7ff6c0072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.900+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/362392565 shutdown_connections 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.900+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/362392565 >> 172.21.15.55:0/362392565 conn(0x7ff6c00b0ef0 msgr2=0x7ff6c00b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.900+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/362392565 shutdown_connections 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.900+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/362392565 wait complete. 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.901+0000 7ff6c6cd6640 10 monclient: build_initial_monmap 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.901+0000 7ff6c6cd6640 10 monclient: monmap: 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:32:54.903 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.901+0000 7ff6c6cd6640 1 librados: starting msgr at 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.901+0000 7ff6c6cd6640 1 librados: starting objecter 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.901+0000 7ff6c6cd6640 1 Processor -- start 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.901+0000 7ff6c6cd6640 1 -- start start 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.901+0000 7ff6c6cd6640 1 librados: setting wanted keys 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.901+0000 7ff6c6cd6640 1 librados: calling monclient init 2024-03-25T18:32:54.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.901+0000 7ff6c6cd6640 10 monclient: init 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c0061830 addr v1:172.21.15.133:6789/0 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c0072aa0 addr v1:172.21.15.55:6790/0 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c013c8c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c013d8b0 con 0x7ff6c013c8c0 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c00b4160 con 0x7ff6c0061830 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c00b3be0 con 0x7ff6c0072aa0 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.902+0000 7ff6c6cd6640 10 monclient(hunting): authenticate will time out at 7579.160231s 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.905+0000 7ff6b7fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7ff6c0072aa0 0x7ff6c0135520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:44340/0 (socket says 172.21.15.55:44340) 2024-03-25T18:32:54.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.905+0000 7ff6b7fff640 1 -- 172.21.15.55:0/2752765456 learned_addr learned my addr 172.21.15.55:0/2752765456 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:32:54.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.906+0000 7ff6b57fa640 10 client.?.objecter ms_handle_connect 0x7ff6c0061830 2024-03-25T18:32:54.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.906+0000 7ff6b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:54.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.906+0000 7ff6b57fa640 10 client.?.objecter ms_handle_connect 0x7ff6c0072aa0 2024-03-25T18:32:54.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.906+0000 7ff6b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:54.974 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.974+0000 7ff6b57fa640 10 client.?.objecter ms_handle_connect 0x7ff6c013c8c0 2024-03-25T18:32:54.974 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.974+0000 7ff6b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:54.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.997+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768098020 0 0) 0x7ff6c013d8b0 con 0x7ff6c013c8c0 2024-03-25T18:32:54.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.997+0000 7ff6b57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:54.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.997+0000 7ff6b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:54.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.997+0000 7ff6b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:54.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.997+0000 7ff6b57fa640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:32:54.998 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:54.997+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ac004eb0 con 0x7ff6c013c8c0 2024-03-25T18:32:55.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.015+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2551093986 0 0) 0x7ff6ac004eb0 con 0x7ff6c013c8c0 2024-03-25T18:32:55.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.015+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6c013d8b0 con 0x7ff6c013c8c0 2024-03-25T18:32:55.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.015+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff6b8001950 con 0x7ff6c013c8c0 2024-03-25T18:32:55.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.015+0000 7ff6b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:55.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.015+0000 7ff6b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:55.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.015+0000 7ff6b57fa640 10 monclient(hunting): dump: 2024-03-25T18:32:55.016 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:55.016 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:55.016 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:55.017 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:55.017 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:55.017 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:55.017 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:55.017 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:55.017 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:55.017 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:55.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.015+0000 7ff6b57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2763273133 0 0) 0x7ff6c013d8b0 con 0x7ff6c013c8c0 2024-03-25T18:32:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 >> v1:172.21.15.55:6790/0 conn(0x7ff6c0072aa0 legacy=0x7ff6c0135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 >> v1:172.21.15.133:6789/0 conn(0x7ff6c0061830 legacy=0x7ff6c013d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 1 monclient: found mon.a 2024-03-25T18:32:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:55.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c00b3d70 con 0x7ff6c013c8c0 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6c6cd6640 5 monclient: authenticate success, global_id 4449 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6c6cd6640 10 monclient: _renew_subs 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6c6cd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c01c6770 con 0x7ff6c013c8c0 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6c6cd6640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6c6cd6640 10 monclient: _renew_subs 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6c6cd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c0135c30 con 0x7ff6c013c8c0 2024-03-25T18:32:55.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.028+0000 7ff6c6cd6640 1 librados: init done 2024-03-25T18:32:55.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.029+0000 7ff6c6cd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:55.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.029+0000 7ff6c6cd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:55.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.029+0000 7ff6c6cd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:55.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.029+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c00b3f50 con 0x7ff6c013c8c0 2024-03-25T18:32:55.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.358+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6b8001d60 con 0x7ff6c013c8c0 2024-03-25T18:32:55.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.358+0000 7ff6b57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:55.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.358+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff6b8004980 con 0x7ff6c013c8c0 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.358+0000 7ff6b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.358+0000 7ff6b57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.358+0000 7ff6b57fa640 10 monclient: dump: 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:32:55.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.358+0000 7ff6b6ffd640 4 set_mon_vals no callback set 2024-03-25T18:32:55.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.359+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7ff6b8004ba0 con 0x7ff6c013c8c0 2024-03-25T18:32:55.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.360+0000 7ff6c4a72640 10 monclient: get_auth_request con 0x7ff6ac06dc70 auth_method 0 2024-03-25T18:32:55.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.360+0000 7ff6b57fa640 10 client.4449.objecter ms_handle_connect 0x7ff6ac06dc70 2024-03-25T18:32:55.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.367+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7ff6b8054700 con 0x7ff6c013c8c0 2024-03-25T18:32:55.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.367+0000 7ff6b57fa640 10 client.4449.objecter ms_dispatch 0x7ff6c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:55.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.367+0000 7ff6b57fa640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:55.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.367+0000 7ff6b57fa640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:55.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.367+0000 7ff6b57fa640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-25T18:32:55.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.367+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6b8050210 con 0x7ff6c013c8c0 2024-03-25T18:32:55.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.367+0000 7ff6b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:55.369 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.367+0000 7ff6b57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:55.476 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:05:05.646 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:55.486 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:05:05.656 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:55.525 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:05:05.695 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:55.547 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:05:05.717 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:55.559 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:05.696 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:55.563 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:05:05.733 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:55.566 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:05.703 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:55.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.591+0000 7ff6c6cd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:55.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.591+0000 7ff6c6cd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:55.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.591+0000 7ff6c6cd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:32:55.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.591+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6c00627b0 con 0x7ff6c013c8c0 2024-03-25T18:32:55.598 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:05.736 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:55.602 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:05:05.772 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:55.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.640+0000 7ff6b57fa640 1 -- 172.21.15.55:0/2752765456 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7ff6b8052140 con 0x7ff6c013c8c0 2024-03-25T18:32:55.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.640+0000 7ff6b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:55.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.640+0000 7ff6b57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:55.641 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:32:55.641 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:55.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 10 librados: watch_flush enter 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 10 librados: watch_flush exit 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 10 client.4449.objecter successfully canceled tick 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 >> v1:172.21.15.55:6800/286536064 conn(0x7ff6ac06dc70 legacy=0x7ff6ac070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 10 monclient: shutdown 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 >> v1:172.21.15.55:6789/0 conn(0x7ff6c013c8c0 legacy=0x7ff6c01e2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 shutdown_connections 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 >> 172.21.15.55:0/2752765456 conn(0x7ff6c00b0ef0 msgr2=0x7ff6c013bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 shutdown_connections 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 1 -- 172.21.15.55:0/2752765456 wait complete. 2024-03-25T18:32:55.644 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:32:55.642+0000 7ff6c6cd6640 1 librados: shutdown 2024-03-25T18:32:55.654 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:02.462 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:02.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.577+0000 7f90a88b3640 10 monclient: get_monmap_and_config 2024-03-25T18:33:02.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.577+0000 7f90a88b3640 10 monclient: build_initial_monmap 2024-03-25T18:33:02.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.577+0000 7f90a88b3640 10 monclient: monmap: 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:33:02.578659+0000 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:33:02.578659+0000 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:33:02.580 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:02.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.578+0000 7f90a88b3640 1 Processor -- start 2024-03-25T18:33:02.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.578+0000 7f90a88b3640 1 -- start start 2024-03-25T18:33:02.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.578+0000 7f90a88b3640 10 monclient: init 2024-03-25T18:33:02.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:02.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:02.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 10 monclient(hunting): picked mon.noname-a con 0x7f90a40a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:02.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 10 monclient(hunting): picked mon.noname-c con 0x7f90a413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:02.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 10 monclient(hunting): picked mon.noname-b con 0x7f90a41430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:02.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a4142520 con 0x7f90a40a98f0 2024-03-25T18:33:02.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a40b28a0 con 0x7f90a413f710 2024-03-25T18:33:02.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a40b2a30 con 0x7f90a41430c0 2024-03-25T18:33:02.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:02.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.579+0000 7f90a88b3640 10 monclient(hunting): authenticate will time out at 7586.836981s 2024-03-25T18:33:02.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.589+0000 7f90a259c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f90a40a98f0 0x7f90a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:34934/0 (socket says 172.21.15.55:34934) 2024-03-25T18:33:02.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.589+0000 7f90a259c640 1 -- 172.21.15.55:0/3609905834 learned_addr learned my addr 172.21.15.55:0/3609905834 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:02.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.596+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804389661 0 0) 0x7f90a4142520 con 0x7f90a40a98f0 2024-03-25T18:33:02.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.596+0000 7f90a159a640 10 monclient(hunting): _negotiate 2024-03-25T18:33:02.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.596+0000 7f90a159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:02.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.596+0000 7f90a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:02.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.596+0000 7f90a159a640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:33:02.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.596+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c003520 con 0x7f90a40a98f0 2024-03-25T18:33:02.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.600+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193793756 0 0) 0x7f90a40b2a30 con 0x7f90a41430c0 2024-03-25T18:33:02.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.600+0000 7f90a159a640 10 monclient(hunting): _negotiate 2024-03-25T18:33:02.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.600+0000 7f90a159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:02.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.600+0000 7f90a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:02.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.600+0000 7f90a159a640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:33:02.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.600+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a4142520 con 0x7f90a41430c0 2024-03-25T18:33:02.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3979434954 0 0) 0x7f908c003520 con 0x7f90a40a98f0 2024-03-25T18:33:02.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 >> v1:172.21.15.55:6790/0 conn(0x7f90a41430c0 legacy=0x7f90a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:02.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 >> v1:172.21.15.133:6789/0 conn(0x7f90a413f710 legacy=0x7f90a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:02.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 10 monclient: _finish_hunting 0 2024-03-25T18:33:02.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 1 monclient: found mon.noname-a 2024-03-25T18:33:02.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a40b44c0 con 0x7f90a40a98f0 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 10 monclient: _finish_auth 0 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 10 monclient: _check_auth_tickets 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9094001690 con 0x7f90a40a98f0 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a159a640 10 monclient: dump: 2024-03-25T18:33:02.608 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a88b3640 5 monclient: authenticate success, global_id 4452 2024-03-25T18:33:02.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.605+0000 7f90a88b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:02.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.619+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90940017f0 con 0x7f90a40a98f0 2024-03-25T18:33:02.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.619+0000 7f90a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:02.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.619+0000 7f90a159a640 1 -- 172.21.15.55:0/3609905834 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9094001c00 con 0x7f90a40a98f0 2024-03-25T18:33:02.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.619+0000 7f90a88b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.619+0000 7f90a88b3640 4 set_mon_vals no callback set 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.619+0000 7f90a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.619+0000 7f90a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.619+0000 7f90a159a640 10 monclient: dump: 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:02.621 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:02.622 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:02.622 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:02.622 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:02.622 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:02.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.620+0000 7f90a88b3640 10 monclient: shutdown 2024-03-25T18:33:02.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.620+0000 7f90a88b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:02.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.620+0000 7f90a88b3640 1 -- 172.21.15.55:0/3609905834 >> v1:172.21.15.55:6789/0 conn(0x7f90a40a98f0 legacy=0x7f90a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:02.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.620+0000 7f90a88b3640 1 -- 172.21.15.55:0/3609905834 shutdown_connections 2024-03-25T18:33:02.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.620+0000 7f90a88b3640 1 -- 172.21.15.55:0/3609905834 >> 172.21.15.55:0/3609905834 conn(0x7f90a40b7ea0 msgr2=0x7f90a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:02.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.620+0000 7f90a88b3640 1 -- 172.21.15.55:0/3609905834 shutdown_connections 2024-03-25T18:33:02.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.620+0000 7f90a88b3640 1 -- 172.21.15.55:0/3609905834 wait complete. 2024-03-25T18:33:02.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.621+0000 7f90a88b3640 10 monclient: build_initial_monmap 2024-03-25T18:33:02.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.621+0000 7f90a88b3640 10 monclient: monmap: 2024-03-25T18:33:02.626 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:02.626 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:02.626 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:33:02.626 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:33:02.626 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:02.626 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:33:02.626 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.621+0000 7f90a88b3640 1 librados: starting msgr at 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.621+0000 7f90a88b3640 1 librados: starting objecter 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.621+0000 7f90a88b3640 1 Processor -- start 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.621+0000 7f90a88b3640 1 -- start start 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.621+0000 7f90a88b3640 1 librados: setting wanted keys 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.621+0000 7f90a88b3640 1 librados: calling monclient init 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.621+0000 7f90a88b3640 10 monclient: init 2024-03-25T18:33:02.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:02.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:02.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 10 monclient(hunting): picked mon.noname-c con 0x7f90a40a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:02.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 10 monclient(hunting): picked mon.noname-b con 0x7f90a413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:02.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 10 monclient(hunting): picked mon.noname-a con 0x7f90a41430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:02.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a41db050 con 0x7f90a41430c0 2024-03-25T18:33:02.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a41dab70 con 0x7f90a413f710 2024-03-25T18:33:02.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a41dad00 con 0x7f90a40a98f0 2024-03-25T18:33:02.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:02.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.622+0000 7f90a88b3640 10 monclient(hunting): authenticate will time out at 7586.879895s 2024-03-25T18:33:02.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.633+0000 7f90a1d9b640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f90a413f710 0x7f90a41d7670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:42740/0 (socket says 172.21.15.55:42740) 2024-03-25T18:33:02.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.633+0000 7f90a1d9b640 1 -- 172.21.15.55:0/1511001672 learned_addr learned my addr 172.21.15.55:0/1511001672 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:02.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.635+0000 7f9092ffd640 10 client.?.objecter ms_handle_connect 0x7f90a413f710 2024-03-25T18:33:02.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.635+0000 7f9092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:02.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.641+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203180324 0 0) 0x7f90a41dab70 con 0x7f90a413f710 2024-03-25T18:33:02.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.641+0000 7f9092ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:02.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.641+0000 7f9092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:02.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.641+0000 7f9092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:02.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.641+0000 7f9092ffd640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:33:02.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.641+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9084004eb0 con 0x7f90a413f710 2024-03-25T18:33:02.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.646+0000 7f9092ffd640 10 client.?.objecter ms_handle_connect 0x7f90a41430c0 2024-03-25T18:33:02.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.646+0000 7f9092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:02.653 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.652+0000 7f9092ffd640 10 client.?.objecter ms_handle_connect 0x7f90a40a98f0 2024-03-25T18:33:02.653 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.652+0000 7f9092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:02.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.657+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 17709134 0 0) 0x7f9084004eb0 con 0x7f90a413f710 2024-03-25T18:33:02.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.657+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90a41dab70 con 0x7f90a413f710 2024-03-25T18:33:02.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.657+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9098001a30 con 0x7f90a413f710 2024-03-25T18:33:02.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.657+0000 7f9092ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:02.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.657+0000 7f9092ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:02.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.657+0000 7f9092ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:02.658 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:02.658 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:02.659 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:02.659 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:02.659 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:02.659 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:02.659 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:02.659 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:02.659 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:02.659 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:02.659 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.657+0000 7f9092ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:02.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.658+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877525514 0 0) 0x7f90a41db050 con 0x7f90a41430c0 2024-03-25T18:33:02.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.658+0000 7f9092ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:02.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.658+0000 7f9092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:02.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.658+0000 7f9092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:02.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.658+0000 7f9092ffd640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:33:02.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.658+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9084004eb0 con 0x7f90a41430c0 2024-03-25T18:33:02.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 217055742 0 0) 0x7f90a41dab70 con 0x7f90a413f710 2024-03-25T18:33:02.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 >> v1:172.21.15.55:6790/0 conn(0x7f90a40a98f0 legacy=0x7f90a41db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:02.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 >> v1:172.21.15.55:6789/0 conn(0x7f90a41430c0 legacy=0x7f90a41393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:02.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 1 monclient: found mon.b 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a40b70a0 con 0x7f90a413f710 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f9092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f90a88b3640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f90a88b3640 10 monclient: _renew_subs 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f90a88b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:02.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a40b72f0 con 0x7f90a413f710 2024-03-25T18:33:02.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f90a88b3640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:02.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f90a88b3640 10 monclient: _renew_subs 2024-03-25T18:33:02.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f90a88b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:02.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a41e32a0 con 0x7f90a413f710 2024-03-25T18:33:02.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.665+0000 7f90a88b3640 1 librados: init done 2024-03-25T18:33:02.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.666+0000 7f90a88b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:02.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.666+0000 7f90a88b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:02.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.666+0000 7f90a88b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:02.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.666+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a40627b0 con 0x7f90a413f710 2024-03-25T18:33:02.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.937+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9098001cb0 con 0x7f90a413f710 2024-03-25T18:33:02.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.937+0000 7f9092ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.937+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9098003a60 con 0x7f90a413f710 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.937+0000 7f9092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.937+0000 7f9092ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.937+0000 7f9092ffd640 10 monclient: dump: 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:02.939 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:02.940 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:02.940 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:02.940 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:02.940 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:02.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.937+0000 7f90a0d99640 4 set_mon_vals no callback set 2024-03-25T18:33:02.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.938+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f9098003c80 con 0x7f90a413f710 2024-03-25T18:33:02.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.939+0000 7f90a259c640 10 monclient: get_auth_request con 0x7f908406de20 auth_method 0 2024-03-25T18:33:02.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.939+0000 7f9092ffd640 10 client.4426.objecter ms_handle_connect 0x7f908406de20 2024-03-25T18:33:02.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.943+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f9098054830 con 0x7f90a413f710 2024-03-25T18:33:02.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.943+0000 7f9092ffd640 10 client.4426.objecter ms_dispatch 0x7f90a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:02.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.943+0000 7f9092ffd640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:02.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.943+0000 7f9092ffd640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:02.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.943+0000 7f9092ffd640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T18:33:02.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.943+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9098050340 con 0x7f90a413f710 2024-03-25T18:33:02.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.943+0000 7f9092ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:02.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:02.943+0000 7f9092ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:03.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.184+0000 7f90a88b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:03.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.185+0000 7f90a88b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:03.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.185+0000 7f90a88b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:03.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.185+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90a41db200 con 0x7f90a413f710 2024-03-25T18:33:03.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.229+0000 7f9092ffd640 1 -- 172.21.15.55:0/1511001672 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f90a41db200 con 0x7f90a413f710 2024-03-25T18:33:03.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.229+0000 7f9092ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:03.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.229+0000 7f9092ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:03.231 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:33:03.231 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 10 librados: watch_flush enter 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 10 librados: watch_flush exit 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 10 client.4426.objecter successfully canceled tick 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 >> v1:172.21.15.55:6800/286536064 conn(0x7f908406de20 legacy=0x7f90840702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 10 monclient: shutdown 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 >> v1:172.21.15.133:6789/0 conn(0x7f90a413f710 legacy=0x7f90a41d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 shutdown_connections 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 >> 172.21.15.55:0/1511001672 conn(0x7f90a40b7ea0 msgr2=0x7f90a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.232+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 shutdown_connections 2024-03-25T18:33:03.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.233+0000 7f90a88b3640 1 -- 172.21.15.55:0/1511001672 wait complete. 2024-03-25T18:33:03.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:03.233+0000 7f90a88b3640 1 librados: shutdown 2024-03-25T18:33:03.245 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:10.051 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:10.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.170+0000 7f4b9e71c640 10 monclient: get_monmap_and_config 2024-03-25T18:33:10.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.170+0000 7f4b9e71c640 10 monclient: build_initial_monmap 2024-03-25T18:33:10.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.170+0000 7f4b9e71c640 10 monclient: monmap: 2024-03-25T18:33:10.171 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:10.171 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:10.172 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:33:10.171438+0000 2024-03-25T18:33:10.172 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:33:10.171438+0000 2024-03-25T18:33:10.172 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:10.172 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:10.172 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:10.172 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:33:10.172 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:33:10.172 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.171+0000 7f4b9e71c640 1 Processor -- start 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.171+0000 7f4b9e71c640 1 -- start start 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.171+0000 7f4b9e71c640 10 monclient: init 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 10 monclient(hunting): picked mon.noname-b con 0x7f4b980a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 10 monclient(hunting): picked mon.noname-c con 0x7f4b9813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 10 monclient(hunting): picked mon.noname-a con 0x7f4b981430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b98142520 con 0x7f4b981430c0 2024-03-25T18:33:10.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980b28a0 con 0x7f4b9813f710 2024-03-25T18:33:10.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980b2a30 con 0x7f4b980a98f0 2024-03-25T18:33:10.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:10.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.172+0000 7f4b9e71c640 10 monclient(hunting): authenticate will time out at 7594.429574s 2024-03-25T18:33:10.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.187+0000 7f4b9ccb9640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f4b981430c0 0x7f4b9814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51630/0 (socket says 172.21.15.55:51630) 2024-03-25T18:33:10.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.187+0000 7f4b9ccb9640 1 -- 172.21.15.55:0/629769405 learned_addr learned my addr 172.21.15.55:0/629769405 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:10.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.213+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126540538 0 0) 0x7f4b980b2a30 con 0x7f4b980a98f0 2024-03-25T18:33:10.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.213+0000 7f4b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:10.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.213+0000 7f4b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:10.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.213+0000 7f4b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:10.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.213+0000 7f4b96ffd640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:33:10.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.213+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b80003520 con 0x7f4b980a98f0 2024-03-25T18:33:10.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.221+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824803010 0 0) 0x7f4b980b28a0 con 0x7f4b9813f710 2024-03-25T18:33:10.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.221+0000 7f4b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:10.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.221+0000 7f4b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:10.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.221+0000 7f4b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:10.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.221+0000 7f4b96ffd640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:33:10.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.221+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b980b2a30 con 0x7f4b9813f710 2024-03-25T18:33:10.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.223+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386947485 0 0) 0x7f4b98142520 con 0x7f4b981430c0 2024-03-25T18:33:10.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.223+0000 7f4b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:10.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.223+0000 7f4b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:10.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.223+0000 7f4b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:10.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.223+0000 7f4b96ffd640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:33:10.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.223+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b980b28a0 con 0x7f4b981430c0 2024-03-25T18:33:10.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.241+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4055013009 0 0) 0x7f4b980b2a30 con 0x7f4b9813f710 2024-03-25T18:33:10.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 >> v1:172.21.15.55:6790/0 conn(0x7f4b980a98f0 legacy=0x7f4b980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:10.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 >> v1:172.21.15.55:6789/0 conn(0x7f4b981430c0 legacy=0x7f4b9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:10.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:10.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 1 monclient: found mon.noname-c 2024-03-25T18:33:10.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:10.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:33:10.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980b44c0 con 0x7f4b9813f710 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4b8c001670 con 0x7f4b9813f710 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b96ffd640 10 monclient: dump: 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:10.244 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:10.245 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:10.245 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:10.245 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:10.245 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:10.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b9e71c640 5 monclient: authenticate success, global_id 4429 2024-03-25T18:33:10.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.242+0000 7f4b9e71c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:10.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.263+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b8c0017d0 con 0x7f4b9813f710 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.263+0000 7f4b96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.263+0000 7f4b96ffd640 1 -- 172.21.15.55:0/629769405 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4b8c001be0 con 0x7f4b9813f710 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.263+0000 7f4b9e71c640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.263+0000 7f4b9e71c640 4 set_mon_vals no callback set 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.263+0000 7f4b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.263+0000 7f4b96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.263+0000 7f4b96ffd640 10 monclient: dump: 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:10.264 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:10.265 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:10.265 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:10.265 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:10.265 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:10.265 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:10.265 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:10.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.264+0000 7f4b9e71c640 10 monclient: shutdown 2024-03-25T18:33:10.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.264+0000 7f4b9e71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:10.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.264+0000 7f4b9e71c640 1 -- 172.21.15.55:0/629769405 >> v1:172.21.15.133:6789/0 conn(0x7f4b9813f710 legacy=0x7f4b981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:10.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.264+0000 7f4b9e71c640 1 -- 172.21.15.55:0/629769405 shutdown_connections 2024-03-25T18:33:10.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.264+0000 7f4b9e71c640 1 -- 172.21.15.55:0/629769405 >> 172.21.15.55:0/629769405 conn(0x7f4b980b7ea0 msgr2=0x7f4b980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:10.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.264+0000 7f4b9e71c640 1 -- 172.21.15.55:0/629769405 shutdown_connections 2024-03-25T18:33:10.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.264+0000 7f4b9e71c640 1 -- 172.21.15.55:0/629769405 wait complete. 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.265+0000 7f4b9e71c640 10 monclient: build_initial_monmap 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.265+0000 7f4b9e71c640 10 monclient: monmap: 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:10.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.265+0000 7f4b9e71c640 1 librados: starting msgr at 2024-03-25T18:33:10.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.265+0000 7f4b9e71c640 1 librados: starting objecter 2024-03-25T18:33:10.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.265+0000 7f4b9e71c640 1 Processor -- start 2024-03-25T18:33:10.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 1 -- start start 2024-03-25T18:33:10.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 1 librados: setting wanted keys 2024-03-25T18:33:10.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 1 librados: calling monclient init 2024-03-25T18:33:10.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 10 monclient: init 2024-03-25T18:33:10.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:10.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:10.269 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 10 monclient(hunting): picked mon.noname-c con 0x7f4b980a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:10.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 10 monclient(hunting): picked mon.noname-b con 0x7f4b9813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:10.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 10 monclient(hunting): picked mon.noname-a con 0x7f4b981430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:10.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9813b180 con 0x7f4b981430c0 2024-03-25T18:33:10.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b981db890 con 0x7f4b9813f710 2024-03-25T18:33:10.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980b7b90 con 0x7f4b980a98f0 2024-03-25T18:33:10.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:10.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.266+0000 7f4b9e71c640 10 monclient(hunting): authenticate will time out at 7594.523986s 2024-03-25T18:33:10.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.275+0000 7f4b9ccb9640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f4b981430c0 0x7f4b981db180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:51710/0 (socket says 172.21.15.55:51710) 2024-03-25T18:33:10.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.275+0000 7f4b9ccb9640 1 -- 172.21.15.55:0/2239870705 learned_addr learned my addr 172.21.15.55:0/2239870705 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:10.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.284+0000 7f4b94ff9640 10 client.?.objecter ms_handle_connect 0x7f4b981430c0 2024-03-25T18:33:10.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.284+0000 7f4b94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:10.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.285+0000 7f4b94ff9640 10 client.?.objecter ms_handle_connect 0x7f4b980a98f0 2024-03-25T18:33:10.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.285+0000 7f4b94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:10.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.292+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680011251 0 0) 0x7f4b9813b180 con 0x7f4b981430c0 2024-03-25T18:33:10.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.292+0000 7f4b94ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:10.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.292+0000 7f4b94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:10.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.292+0000 7f4b94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:10.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.292+0000 7f4b94ff9640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:33:10.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.292+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b84004eb0 con 0x7f4b981430c0 2024-03-25T18:33:10.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.293+0000 7f4b94ff9640 10 client.?.objecter ms_handle_connect 0x7f4b9813f710 2024-03-25T18:33:10.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.293+0000 7f4b94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:10.302 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.302+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254050599 0 0) 0x7f4b980b7b90 con 0x7f4b980a98f0 2024-03-25T18:33:10.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.302+0000 7f4b94ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:10.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.302+0000 7f4b94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:10.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.302+0000 7f4b94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:10.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.302+0000 7f4b94ff9640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:33:10.303 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.302+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b9813b180 con 0x7f4b980a98f0 2024-03-25T18:33:10.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.305+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585982364 0 0) 0x7f4b981db890 con 0x7f4b9813f710 2024-03-25T18:33:10.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.305+0000 7f4b94ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:10.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.305+0000 7f4b94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:10.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.305+0000 7f4b94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:10.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.305+0000 7f4b94ff9640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:33:10.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.305+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b980b7b90 con 0x7f4b9813f710 2024-03-25T18:33:10.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.309+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 716656519 0 0) 0x7f4b84004eb0 con 0x7f4b981430c0 2024-03-25T18:33:10.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.309+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b981db890 con 0x7f4b981430c0 2024-03-25T18:33:10.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.309+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4b90001bd0 con 0x7f4b981430c0 2024-03-25T18:33:10.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.309+0000 7f4b94ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.309+0000 7f4b94ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.309+0000 7f4b94ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:10.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.309+0000 7f4b94ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:10.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.324+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1962058878 0 0) 0x7f4b981db890 con 0x7f4b981430c0 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.324+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 >> v1:172.21.15.55:6790/0 conn(0x7f4b980a98f0 legacy=0x7f4b980b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.324+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 >> v1:172.21.15.133:6789/0 conn(0x7f4b9813f710 legacy=0x7f4b981daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.324+0000 7f4b94ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.324+0000 7f4b94ff9640 1 monclient: found mon.a 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b94ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b94ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9813b3f0 con 0x7f4b981430c0 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b94ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b94ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b9e71c640 5 monclient: authenticate success, global_id 4461 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b9e71c640 10 monclient: _renew_subs 2024-03-25T18:33:10.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b9e71c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9813a1b0 con 0x7f4b981430c0 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b9e71c640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b9e71c640 10 monclient: _renew_subs 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b9e71c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9813ba40 con 0x7f4b981430c0 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.325+0000 7f4b9e71c640 1 librados: init done 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.326+0000 7f4b9e71c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.326+0000 7f4b9e71c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.326+0000 7f4b9e71c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:10.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.326+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b980b3ef0 con 0x7f4b981430c0 2024-03-25T18:33:10.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.646+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b90002080 con 0x7f4b981430c0 2024-03-25T18:33:10.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.646+0000 7f4b94ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:10.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.646+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4b90004d00 con 0x7f4b981430c0 2024-03-25T18:33:10.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.646+0000 7f4b94ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:10.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.646+0000 7f4b94ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:10.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.646+0000 7f4b94ff9640 10 monclient: dump: 2024-03-25T18:33:10.647 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:10.647 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:10.647 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:10.648 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:10.648 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:10.648 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:10.648 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:10.648 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:10.648 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:10.648 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:10.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.646+0000 7f4b967fc640 4 set_mon_vals no callback set 2024-03-25T18:33:10.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.647+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f4b90004f20 con 0x7f4b981430c0 2024-03-25T18:33:10.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.647+0000 7f4b97fff640 10 monclient: get_auth_request con 0x7f4b8406dd00 auth_method 0 2024-03-25T18:33:10.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.647+0000 7f4b94ff9640 10 client.4461.objecter ms_handle_connect 0x7f4b8406dd00 2024-03-25T18:33:10.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.655+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f4b90054b60 con 0x7f4b981430c0 2024-03-25T18:33:10.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.655+0000 7f4b94ff9640 10 client.4461.objecter ms_dispatch 0x7f4b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:10.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.655+0000 7f4b94ff9640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:10.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.655+0000 7f4b94ff9640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:10.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.655+0000 7f4b94ff9640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-25T18:33:10.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.655+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b900504a0 con 0x7f4b981430c0 2024-03-25T18:33:10.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.655+0000 7f4b94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:10.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.655+0000 7f4b94ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:10.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.882+0000 7f4b9e71c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:10.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.882+0000 7f4b9e71c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:10.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.882+0000 7f4b9e71c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:10.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.882+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b980627b0 con 0x7f4b981430c0 2024-03-25T18:33:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.904+0000 7f4b94ff9640 1 -- 172.21.15.55:0/2239870705 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f4b900525a0 con 0x7f4b981430c0 2024-03-25T18:33:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.905+0000 7f4b94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:10.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.905+0000 7f4b94ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:10.906 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:33:10.906 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 10 librados: watch_flush enter 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 10 librados: watch_flush exit 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 10 client.4461.objecter successfully canceled tick 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 >> v1:172.21.15.55:6800/286536064 conn(0x7f4b8406dd00 legacy=0x7f4b840701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 10 monclient: shutdown 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 >> v1:172.21.15.55:6789/0 conn(0x7f4b981430c0 legacy=0x7f4b981db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 shutdown_connections 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.907+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 >> 172.21.15.55:0/2239870705 conn(0x7f4b980b7ea0 msgr2=0x7f4b98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.908+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 shutdown_connections 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.908+0000 7f4b9e71c640 1 -- 172.21.15.55:0/2239870705 wait complete. 2024-03-25T18:33:10.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:10.908+0000 7f4b9e71c640 1 librados: shutdown 2024-03-25T18:33:10.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:15.734 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:05:25.904 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:15.739 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:05:25.909 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:15.772 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:05:25.942 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:15.819 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:05:25.989 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:15.825 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:25.962 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:15.829 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:05:25.999 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:15.837 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:25.974 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:15.861 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:05:26.031 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:15.871 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:26.008 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:17.726 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.841+0000 7f822f4d3640 10 monclient: get_monmap_and_config 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.841+0000 7f822f4d3640 10 monclient: build_initial_monmap 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.841+0000 7f822f4d3640 10 monclient: monmap: 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:33:17.841988+0000 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:33:17.841988+0000 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:17.842 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:33:17.843 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:33:17.843 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.841+0000 7f822f4d3640 1 Processor -- start 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 1 -- start start 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 10 monclient: init 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f82280705b0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f82281402d0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f8228149ae0 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82280718c0 con 0x7f82281402d0 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82280714c0 con 0x7f8228149ae0 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8228072b50 con 0x7f82280705b0 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:17.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.842+0000 7f822f4d3640 10 monclient(hunting): authenticate will time out at 7602.100292s 2024-03-25T18:33:17.852 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.852+0000 7f822d26f640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f82280705b0 0x7f822813d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:44378/0 (socket says 172.21.15.55:44378) 2024-03-25T18:33:17.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.852+0000 7f822d26f640 1 -- 172.21.15.55:0/3837576918 learned_addr learned my addr 172.21.15.55:0/3837576918 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.865+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757905547 0 0) 0x7f82280714c0 con 0x7f8228149ae0 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.865+0000 7f821ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.865+0000 7f821ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.865+0000 7f821ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.865+0000 7f821ffff640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:33:17.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.865+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8214003520 con 0x7f8228149ae0 2024-03-25T18:33:17.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.877+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925765927 0 0) 0x7f8228072b50 con 0x7f82280705b0 2024-03-25T18:33:17.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.877+0000 7f821ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:17.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.877+0000 7f821ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:17.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.877+0000 7f821ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:17.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.877+0000 7f821ffff640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:33:17.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.877+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82280714c0 con 0x7f82280705b0 2024-03-25T18:33:17.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 512565277 0 0) 0x7f8214003520 con 0x7f8228149ae0 2024-03-25T18:33:17.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 >> v1:172.21.15.55:6790/0 conn(0x7f82280705b0 legacy=0x7f822813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 >> v1:172.21.15.55:6789/0 conn(0x7f82281402d0 legacy=0x7f8228140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:17.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 1 monclient: found mon.noname-c 2024-03-25T18:33:17.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:17.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822813dd10 con 0x7f8228149ae0 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 10 monclient: _finish_auth 0 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8220001a10 con 0x7f8228149ae0 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f822f4d3640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f822f4d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f821ffff640 10 monclient: dump: 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:17.882 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:17.883 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:17.883 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:17.883 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:17.883 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:17.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.880+0000 7f822f4d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.889+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8220001bf0 con 0x7f8228149ae0 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.889+0000 7f821ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.889+0000 7f821ffff640 1 -- 172.21.15.55:0/3837576918 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8220002000 con 0x7f8228149ae0 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.889+0000 7f822f4d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.889+0000 7f822f4d3640 4 set_mon_vals no callback set 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.889+0000 7f821ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.889+0000 7f821ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.889+0000 7f821ffff640 10 monclient: dump: 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:17.890 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:17.891 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:17.891 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:17.891 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:17.891 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:17.891 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:17.891 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:17.891 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:17.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.890+0000 7f822f4d3640 10 monclient: shutdown 2024-03-25T18:33:17.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.890+0000 7f822f4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:17.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.890+0000 7f822f4d3640 1 -- 172.21.15.55:0/3837576918 >> v1:172.21.15.133:6789/0 conn(0x7f8228149ae0 legacy=0x7f822814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.890+0000 7f822f4d3640 1 -- 172.21.15.55:0/3837576918 shutdown_connections 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.890+0000 7f822f4d3640 1 -- 172.21.15.55:0/3837576918 >> 172.21.15.55:0/3837576918 conn(0x7f82280b5d10 msgr2=0x7f82280b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.890+0000 7f822f4d3640 1 -- 172.21.15.55:0/3837576918 shutdown_connections 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.890+0000 7f822f4d3640 1 -- 172.21.15.55:0/3837576918 wait complete. 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.891+0000 7f822f4d3640 10 monclient: build_initial_monmap 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.891+0000 7f822f4d3640 10 monclient: monmap: 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:33:17.893 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:33:17.894 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:17.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.891+0000 7f822f4d3640 1 librados: starting msgr at 2024-03-25T18:33:17.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.891+0000 7f822f4d3640 1 librados: starting objecter 2024-03-25T18:33:17.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.891+0000 7f822f4d3640 1 Processor -- start 2024-03-25T18:33:17.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 1 -- start start 2024-03-25T18:33:17.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 1 librados: setting wanted keys 2024-03-25T18:33:17.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 1 librados: calling monclient init 2024-03-25T18:33:17.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 10 monclient: init 2024-03-25T18:33:17.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:17.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:17.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f82280705b0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:17.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f82281402d0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:17.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f8228149ae0 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:17.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8228146690 con 0x7f82281402d0 2024-03-25T18:33:17.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82281db960 con 0x7f8228149ae0 2024-03-25T18:33:17.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822813abe0 con 0x7f82280705b0 2024-03-25T18:33:17.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:17.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.892+0000 7f822f4d3640 10 monclient(hunting): authenticate will time out at 7602.150284s 2024-03-25T18:33:17.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.897+0000 7f822da70640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f8228149ae0 0x7f8228139ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:37288/0 (socket says 172.21.15.55:37288) 2024-03-25T18:33:17.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.897+0000 7f822da70640 1 -- 172.21.15.55:0/3829800258 learned_addr learned my addr 172.21.15.55:0/3829800258 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:17.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.899+0000 7f821dffb640 10 client.?.objecter ms_handle_connect 0x7f8228149ae0 2024-03-25T18:33:17.900 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.899+0000 7f821dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:17.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.908+0000 7f821dffb640 10 client.?.objecter ms_handle_connect 0x7f82281402d0 2024-03-25T18:33:17.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.908+0000 7f821dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:17.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.909+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114776323 0 0) 0x7f82281db960 con 0x7f8228149ae0 2024-03-25T18:33:17.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.909+0000 7f821dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:17.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.909+0000 7f821dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:17.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.909+0000 7f821dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:17.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.909+0000 7f821dffb640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:33:17.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.909+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f820c004eb0 con 0x7f8228149ae0 2024-03-25T18:33:17.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.912+0000 7f821dffb640 10 client.?.objecter ms_handle_connect 0x7f82280705b0 2024-03-25T18:33:17.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.912+0000 7f821dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:17.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.924+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573475090 0 0) 0x7f8228146690 con 0x7f82281402d0 2024-03-25T18:33:17.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.924+0000 7f821dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:17.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.924+0000 7f821dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:17.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.924+0000 7f821dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:17.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.924+0000 7f821dffb640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:33:17.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.924+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82281db960 con 0x7f82281402d0 2024-03-25T18:33:17.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.927+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337093505 0 0) 0x7f822813abe0 con 0x7f82280705b0 2024-03-25T18:33:17.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.927+0000 7f821dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:33:17.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.927+0000 7f821dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:17.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.927+0000 7f821dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:17.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.927+0000 7f821dffb640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:33:17.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.927+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8228146690 con 0x7f82280705b0 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.929+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2228482634 0 0) 0x7f820c004eb0 con 0x7f8228149ae0 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.929+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f822813abe0 con 0x7f8228149ae0 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.929+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8220001f40 con 0x7f8228149ae0 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.929+0000 7f821dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.929+0000 7f821dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.929+0000 7f821dffb640 10 monclient(hunting): dump: 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:17.931 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:17.932 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:17.932 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:17.932 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:17.932 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:17.932 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:17.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.929+0000 7f821dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:17.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.938+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1503554489 0 0) 0x7f822813abe0 con 0x7f8228149ae0 2024-03-25T18:33:17.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.938+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 >> v1:172.21.15.55:6790/0 conn(0x7f82280705b0 legacy=0x7f82280adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.938+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 >> v1:172.21.15.55:6789/0 conn(0x7f82281402d0 legacy=0x7f822806d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.938+0000 7f821dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f821dffb640 1 monclient: found mon.b 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f821dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f821dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82281dbe80 con 0x7f8228149ae0 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f821dffb640 10 monclient: _finish_auth 0 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f821dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f821dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f822f4d3640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f822f4d3640 10 monclient: _renew_subs 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f822f4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:17.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82280ae210 con 0x7f8228149ae0 2024-03-25T18:33:17.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f822f4d3640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:17.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f822f4d3640 10 monclient: _renew_subs 2024-03-25T18:33:17.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f822f4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:17.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82281db540 con 0x7f8228149ae0 2024-03-25T18:33:17.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.939+0000 7f822f4d3640 1 librados: init done 2024-03-25T18:33:17.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.940+0000 7f822f4d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:17.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.940+0000 7f822f4d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:17.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.940+0000 7f822f4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:17.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:17.940+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82281dc070 con 0x7f8228149ae0 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.218+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8220003c20 con 0x7f8228149ae0 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.218+0000 7f821dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.218+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f82200015d0 con 0x7f8228149ae0 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.218+0000 7f821dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.218+0000 7f821dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.218+0000 7f821dffb640 10 monclient: dump: 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:18.220 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:18.221 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:18.221 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:18.221 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:18.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.218+0000 7f821f7fe640 4 set_mon_vals no callback set 2024-03-25T18:33:18.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.219+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f82200040b0 con 0x7f8228149ae0 2024-03-25T18:33:18.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.220+0000 7f822d26f640 10 monclient: get_auth_request con 0x7f820c06ddf0 auth_method 0 2024-03-25T18:33:18.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.220+0000 7f821dffb640 10 client.4438.objecter ms_handle_connect 0x7f820c06ddf0 2024-03-25T18:33:18.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.224+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f8220054b80 con 0x7f8228149ae0 2024-03-25T18:33:18.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.224+0000 7f821dffb640 10 client.4438.objecter ms_dispatch 0x7f8228000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:18.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.224+0000 7f821dffb640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:18.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.224+0000 7f821dffb640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:18.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.224+0000 7f821dffb640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-25T18:33:18.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.224+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8220050690 con 0x7f8228149ae0 2024-03-25T18:33:18.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.224+0000 7f821dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:18.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.224+0000 7f821dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:18.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.485+0000 7f822f4d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:18.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.485+0000 7f822f4d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:18.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.485+0000 7f822f4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:18.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.485+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82281dbaf0 con 0x7f8228149ae0 2024-03-25T18:33:18.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.541+0000 7f821dffb640 1 -- 172.21.15.55:0/3829800258 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f82200525c0 con 0x7f8228149ae0 2024-03-25T18:33:18.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.541+0000 7f821dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:18.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.541+0000 7f821dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:18.542 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:33:18.542 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:18.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 10 librados: watch_flush enter 2024-03-25T18:33:18.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 10 librados: watch_flush exit 2024-03-25T18:33:18.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-25T18:33:18.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 10 client.4438.objecter successfully canceled tick 2024-03-25T18:33:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 >> v1:172.21.15.55:6800/286536064 conn(0x7f820c06ddf0 legacy=0x7f820c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 10 monclient: shutdown 2024-03-25T18:33:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 >> v1:172.21.15.133:6789/0 conn(0x7f8228149ae0 legacy=0x7f8228139ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 shutdown_connections 2024-03-25T18:33:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 >> 172.21.15.55:0/3829800258 conn(0x7f82280b5d10 msgr2=0x7f82280b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 shutdown_connections 2024-03-25T18:33:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.544+0000 7f822f4d3640 1 -- 172.21.15.55:0/3829800258 wait complete. 2024-03-25T18:33:18.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:18.545+0000 7f822f4d3640 1 librados: shutdown 2024-03-25T18:33:18.556 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:25.361 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:25.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.476+0000 7f7fb4b8f640 10 monclient: get_monmap_and_config 2024-03-25T18:33:25.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.476+0000 7f7fb4b8f640 10 monclient: build_initial_monmap 2024-03-25T18:33:25.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.476+0000 7f7fb4b8f640 10 monclient: monmap: 2024-03-25T18:33:25.477 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:25.477 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:25.477 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:33:25.477449+0000 2024-03-25T18:33:25.478 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:33:25.477449+0000 2024-03-25T18:33:25.478 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:25.478 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:25.478 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:25.478 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:33:25.478 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:33:25.478 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:25.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.477+0000 7f7fb4b8f640 1 Processor -- start 2024-03-25T18:33:25.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.477+0000 7f7fb4b8f640 1 -- start start 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.477+0000 7f7fb4b8f640 10 monclient: init 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 10 monclient(hunting): picked mon.noname-b con 0x7f7fb00a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 10 monclient(hunting): picked mon.noname-a con 0x7f7fb013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 10 monclient(hunting): picked mon.noname-c con 0x7f7fb01430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb0142520 con 0x7f7fb013f710 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb00b28a0 con 0x7f7fb01430c0 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb00b2a30 con 0x7f7fb00a98f0 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:25.480 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.478+0000 7f7fb4b8f640 10 monclient(hunting): authenticate will time out at 7609.735699s 2024-03-25T18:33:25.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.492+0000 7f7faed9d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f7fb01430c0 0x7f7fb014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:58274/0 (socket says 172.21.15.55:58274) 2024-03-25T18:33:25.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.492+0000 7f7faed9d640 1 -- 172.21.15.55:0/2460624309 learned_addr learned my addr 172.21.15.55:0/2460624309 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:25.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.500+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522314139 0 0) 0x7f7fb0142520 con 0x7f7fb013f710 2024-03-25T18:33:25.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.500+0000 7f7fadd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:25.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.500+0000 7f7fadd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:25.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.500+0000 7f7fadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:25.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.500+0000 7f7fadd9b640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:33:25.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.501+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9c003520 con 0x7f7fb013f710 2024-03-25T18:33:25.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.508+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865907589 0 0) 0x7f7fb00b2a30 con 0x7f7fb00a98f0 2024-03-25T18:33:25.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.508+0000 7f7fadd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:25.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.508+0000 7f7fadd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:25.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.508+0000 7f7fadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:25.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.508+0000 7f7fadd9b640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:33:25.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.508+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0142520 con 0x7f7fb00a98f0 2024-03-25T18:33:25.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.512+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585993934 0 0) 0x7f7fb00b28a0 con 0x7f7fb01430c0 2024-03-25T18:33:25.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.512+0000 7f7fadd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:25.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.512+0000 7f7fadd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:25.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.512+0000 7f7fadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:25.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.512+0000 7f7fadd9b640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:33:25.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.512+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb00b2a30 con 0x7f7fb01430c0 2024-03-25T18:33:25.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1604072147 0 0) 0x7f7f9c003520 con 0x7f7fb013f710 2024-03-25T18:33:25.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 >> v1:172.21.15.55:6790/0 conn(0x7f7fb00a98f0 legacy=0x7f7fb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 >> v1:172.21.15.133:6789/0 conn(0x7f7fb01430c0 legacy=0x7f7fb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 1 monclient: found mon.noname-a 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb00b44c0 con 0x7f7fb013f710 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7f98001690 con 0x7f7fb013f710 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fb4b8f640 5 monclient: authenticate success, global_id 4470 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fb4b8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:25.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fadd9b640 10 monclient: dump: 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:25.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.520+0000 7f7fb4b8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.540+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f980017f0 con 0x7f7fb013f710 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.540+0000 7f7fadd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.540+0000 7f7fadd9b640 1 -- 172.21.15.55:0/2460624309 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7f98001c00 con 0x7f7fb013f710 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.540+0000 7f7fadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.540+0000 7f7fadd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.540+0000 7f7fadd9b640 10 monclient: dump: 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.541 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.542 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:25.542 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:25.542 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:25.542 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:25.542 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:25.542 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:25.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.540+0000 7f7fb4b8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:25.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.540+0000 7f7fb4b8f640 4 set_mon_vals no callback set 2024-03-25T18:33:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.541+0000 7f7fb4b8f640 10 monclient: shutdown 2024-03-25T18:33:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.541+0000 7f7fb4b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.541+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/2460624309 >> v1:172.21.15.55:6789/0 conn(0x7f7fb013f710 legacy=0x7f7fb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.541+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/2460624309 shutdown_connections 2024-03-25T18:33:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.541+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/2460624309 >> 172.21.15.55:0/2460624309 conn(0x7f7fb00b7ea0 msgr2=0x7f7fb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.541+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/2460624309 shutdown_connections 2024-03-25T18:33:25.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.541+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/2460624309 wait complete. 2024-03-25T18:33:25.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.542+0000 7f7fb4b8f640 10 monclient: build_initial_monmap 2024-03-25T18:33:25.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.542+0000 7f7fb4b8f640 10 monclient: monmap: 2024-03-25T18:33:25.544 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:25.544 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:25.544 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:33:25.544 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:33:25.544 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:25.544 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:33:25.544 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.543+0000 7f7fb4b8f640 1 librados: starting msgr at 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.543+0000 7f7fb4b8f640 1 librados: starting objecter 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.543+0000 7f7fb4b8f640 1 Processor -- start 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.543+0000 7f7fb4b8f640 1 -- start start 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.543+0000 7f7fb4b8f640 1 librados: setting wanted keys 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.543+0000 7f7fb4b8f640 1 librados: calling monclient init 2024-03-25T18:33:25.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.543+0000 7f7fb4b8f640 10 monclient: init 2024-03-25T18:33:25.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:25.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:25.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 10 monclient(hunting): picked mon.noname-b con 0x7f7fb00a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 10 monclient(hunting): picked mon.noname-c con 0x7f7fb013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 10 monclient(hunting): picked mon.noname-a con 0x7f7fb01430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb0137080 con 0x7f7fb01430c0 2024-03-25T18:33:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb0137360 con 0x7f7fb00a98f0 2024-03-25T18:33:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb0135b10 con 0x7f7fb013f710 2024-03-25T18:33:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:25.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.544+0000 7f7fb4b8f640 10 monclient(hunting): authenticate will time out at 7609.802088s 2024-03-25T18:33:25.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.551+0000 7f7faed9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f7fb01430c0 0x7f7fb0130340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:50352/0 (socket says 172.21.15.55:50352) 2024-03-25T18:33:25.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.551+0000 7f7faed9d640 1 -- 172.21.15.55:0/4060638262 learned_addr learned my addr 172.21.15.55:0/4060638262 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:25.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.553+0000 7f7fa6ffd640 10 client.?.objecter ms_handle_connect 0x7f7fb01430c0 2024-03-25T18:33:25.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.553+0000 7f7fa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.555+0000 7f7fa6ffd640 10 client.?.objecter ms_handle_connect 0x7f7fb00a98f0 2024-03-25T18:33:25.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.555+0000 7f7fa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.557+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127521564 0 0) 0x7f7fb0137360 con 0x7f7fb00a98f0 2024-03-25T18:33:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.557+0000 7f7fa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:25.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.557+0000 7f7fa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.557+0000 7f7fa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.557+0000 7f7fa6ffd640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:33:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.557+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f90004eb0 con 0x7f7fb00a98f0 2024-03-25T18:33:25.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.559+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433713465 0 0) 0x7f7fb0137080 con 0x7f7fb01430c0 2024-03-25T18:33:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.559+0000 7f7fa6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.559+0000 7f7fa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.559+0000 7f7fa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.559+0000 7f7fa6ffd640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:33:25.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.559+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0137360 con 0x7f7fb01430c0 2024-03-25T18:33:25.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.564+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2329318836 0 0) 0x7f7f90004eb0 con 0x7f7fb00a98f0 2024-03-25T18:33:25.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.564+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fb0137080 con 0x7f7fb00a98f0 2024-03-25T18:33:25.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.564+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7fa0001970 con 0x7f7fb00a98f0 2024-03-25T18:33:25.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.564+0000 7f7fa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:25.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.564+0000 7f7fa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:25.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.564+0000 7f7fa6ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:25.565 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:25.565 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.564+0000 7f7fa6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.564+0000 7f7fa6ffd640 10 client.?.objecter ms_handle_connect 0x7f7fb013f710 2024-03-25T18:33:25.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.564+0000 7f7fa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:25.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.567+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1975728912 0 0) 0x7f7fb0137360 con 0x7f7fb01430c0 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.567+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f90004eb0 con 0x7f7fb01430c0 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.567+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7fa8001bd0 con 0x7f7fb01430c0 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.567+0000 7f7fa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.567+0000 7f7fa6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.567+0000 7f7fa6ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:25.569 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:25.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2504689175 0 0) 0x7f7fb0137080 con 0x7f7fb00a98f0 2024-03-25T18:33:25.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 >> v1:172.21.15.55:6790/0 conn(0x7f7fb013f710 legacy=0x7f7fb00b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:25.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 >> v1:172.21.15.55:6789/0 conn(0x7f7fb01430c0 legacy=0x7f7fb0130340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:25.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:25.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 1 monclient: found mon.b 2024-03-25T18:33:25.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:25.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:25.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb0135df0 con 0x7f7fb00a98f0 2024-03-25T18:33:25.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fb4b8f640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fb4b8f640 10 monclient: _renew_subs 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fb4b8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb0136340 con 0x7f7fb00a98f0 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fb4b8f640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fb4b8f640 10 monclient: _renew_subs 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fb4b8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb01352e0 con 0x7f7fb00a98f0 2024-03-25T18:33:25.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.571+0000 7f7fb4b8f640 1 librados: init done 2024-03-25T18:33:25.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.572+0000 7f7fb4b8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:25.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.572+0000 7f7fb4b8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:25.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.572+0000 7f7fb4b8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:25.574 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.572+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb00627b0 con 0x7f7fb00a98f0 2024-03-25T18:33:25.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.574+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fa00035f0 con 0x7f7fb00a98f0 2024-03-25T18:33:25.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.574+0000 7f7fa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:25.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.574+0000 7f7fad59a640 4 set_mon_vals no callback set 2024-03-25T18:33:25.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.853+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7fa0003900 con 0x7f7fb00a98f0 2024-03-25T18:33:25.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.853+0000 7f7fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:25.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.853+0000 7f7fa6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.853+0000 7f7fa6ffd640 10 monclient: dump: 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:25.855 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:25.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.854+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f7fa0019950 con 0x7f7fb00a98f0 2024-03-25T18:33:25.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.854+0000 7f7fa7fff640 10 monclient: get_auth_request con 0x7f7f9006de00 auth_method 0 2024-03-25T18:33:25.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.855+0000 7f7fa6ffd640 10 client.4444.objecter ms_handle_connect 0x7f7f9006de00 2024-03-25T18:33:25.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.857+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f7fa0053670 con 0x7f7fb00a98f0 2024-03-25T18:33:25.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.857+0000 7f7fa6ffd640 10 client.4444.objecter ms_dispatch 0x7f7fb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:25.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.857+0000 7f7fa6ffd640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:25.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.857+0000 7f7fa6ffd640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:25.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.857+0000 7f7fa6ffd640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-25T18:33:25.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.860+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fa00b08b0 con 0x7f7fb00a98f0 2024-03-25T18:33:25.860 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.860+0000 7f7fa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:25.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:25.860+0000 7f7fa6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:26.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.106+0000 7f7fb4b8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:26.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.106+0000 7f7fb4b8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:26.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.106+0000 7f7fb4b8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:26.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.106+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fb0074cc0 con 0x7f7fb00a98f0 2024-03-25T18:33:26.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.231+0000 7f7fa6ffd640 1 -- 172.21.15.55:0/4060638262 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f7fa004efb0 con 0x7f7fb00a98f0 2024-03-25T18:33:26.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.231+0000 7f7fa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:26.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.231+0000 7f7fa6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:26.232 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:33:26.232 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:26.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.233+0000 7f7fb4b8f640 10 librados: watch_flush enter 2024-03-25T18:33:26.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 10 librados: watch_flush exit 2024-03-25T18:33:26.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-25T18:33:26.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 10 client.4444.objecter successfully canceled tick 2024-03-25T18:33:26.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 >> v1:172.21.15.55:6800/286536064 conn(0x7f7f9006de00 legacy=0x7f7f900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:26.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 10 monclient: shutdown 2024-03-25T18:33:26.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:26.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 >> v1:172.21.15.133:6789/0 conn(0x7f7fb00a98f0 legacy=0x7f7fb0137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:26.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 shutdown_connections 2024-03-25T18:33:26.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 >> 172.21.15.55:0/4060638262 conn(0x7f7fb00b7ea0 msgr2=0x7f7fb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:26.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 shutdown_connections 2024-03-25T18:33:26.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 1 -- 172.21.15.55:0/4060638262 wait complete. 2024-03-25T18:33:26.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:26.234+0000 7f7fb4b8f640 1 librados: shutdown 2024-03-25T18:33:26.246 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:33.053 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.171+0000 7efd44754640 10 monclient: get_monmap_and_config 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.171+0000 7efd44754640 10 monclient: build_initial_monmap 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.171+0000 7efd44754640 10 monclient: monmap: 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:33:33.172946+0000 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:33:33.172946+0000 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:33:33.173 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.171+0000 7efd44754640 1 Processor -- start 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.171+0000 7efd44754640 1 -- start start 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.171+0000 7efd44754640 10 monclient: init 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 10 monclient(hunting): picked mon.noname-a con 0x7efd3c0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 10 monclient(hunting): picked mon.noname-b con 0x7efd3c13f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 10 monclient(hunting): picked mon.noname-c con 0x7efd3c1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c142520 con 0x7efd3c0a98f0 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0b28a0 con 0x7efd3c1430c0 2024-03-25T18:33:33.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0b2a30 con 0x7efd3c13f710 2024-03-25T18:33:33.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:33.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.172+0000 7efd44754640 10 monclient(hunting): authenticate will time out at 7617.431274s 2024-03-25T18:33:33.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.180+0000 7efd41cef640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7efd3c13f710 0x7efd3c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:44804/0 (socket says 172.21.15.55:44804) 2024-03-25T18:33:33.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.180+0000 7efd41cef640 1 -- 172.21.15.55:0/2477165313 learned_addr learned my addr 172.21.15.55:0/2477165313 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:33.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.200+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492475757 0 0) 0x7efd3c142520 con 0x7efd3c0a98f0 2024-03-25T18:33:33.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.200+0000 7efd414ee640 10 monclient(hunting): _negotiate 2024-03-25T18:33:33.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.200+0000 7efd414ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:33.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.200+0000 7efd414ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:33.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.200+0000 7efd414ee640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:33:33.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.200+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd28003520 con 0x7efd3c0a98f0 2024-03-25T18:33:33.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.203+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420886411 0 0) 0x7efd3c0b2a30 con 0x7efd3c13f710 2024-03-25T18:33:33.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.203+0000 7efd414ee640 10 monclient(hunting): _negotiate 2024-03-25T18:33:33.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.203+0000 7efd414ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:33.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.203+0000 7efd414ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:33.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.203+0000 7efd414ee640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:33:33.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.203+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c142520 con 0x7efd3c13f710 2024-03-25T18:33:33.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.217+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751400196 0 0) 0x7efd3c0b28a0 con 0x7efd3c1430c0 2024-03-25T18:33:33.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.218+0000 7efd414ee640 10 monclient(hunting): _negotiate 2024-03-25T18:33:33.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.218+0000 7efd414ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:33.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.218+0000 7efd414ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:33.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.218+0000 7efd414ee640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:33:33.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.218+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c0b2a30 con 0x7efd3c1430c0 2024-03-25T18:33:33.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3610366883 0 0) 0x7efd3c142520 con 0x7efd3c13f710 2024-03-25T18:33:33.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 >> v1:172.21.15.133:6789/0 conn(0x7efd3c1430c0 legacy=0x7efd3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 >> v1:172.21.15.55:6789/0 conn(0x7efd3c0a98f0 legacy=0x7efd3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 10 monclient: _finish_hunting 0 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 1 monclient: found mon.noname-b 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c0b44c0 con 0x7efd3c13f710 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 10 monclient: _finish_auth 0 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 10 monclient: _check_auth_tickets 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efd2c001670 con 0x7efd3c13f710 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd44754640 5 monclient: authenticate success, global_id 4463 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd44754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:33.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd414ee640 10 monclient: dump: 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:33.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.219+0000 7efd44754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.227+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd2c0017d0 con 0x7efd3c13f710 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.227+0000 7efd414ee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.227+0000 7efd414ee640 1 -- 172.21.15.55:0/2477165313 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efd2c001be0 con 0x7efd3c13f710 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.227+0000 7efd414ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.227+0000 7efd414ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.227+0000 7efd414ee640 10 monclient: dump: 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.229 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.230 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:33.230 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:33.230 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:33.230 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:33.230 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:33.230 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:33.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.227+0000 7efd44754640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:33.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.227+0000 7efd44754640 4 set_mon_vals no callback set 2024-03-25T18:33:33.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.228+0000 7efd44754640 10 monclient: shutdown 2024-03-25T18:33:33.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.228+0000 7efd44754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:33.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.228+0000 7efd44754640 1 -- 172.21.15.55:0/2477165313 >> v1:172.21.15.55:6790/0 conn(0x7efd3c13f710 legacy=0x7efd3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:33.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.228+0000 7efd44754640 1 -- 172.21.15.55:0/2477165313 shutdown_connections 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.228+0000 7efd44754640 1 -- 172.21.15.55:0/2477165313 >> 172.21.15.55:0/2477165313 conn(0x7efd3c0b7ea0 msgr2=0x7efd3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.228+0000 7efd44754640 1 -- 172.21.15.55:0/2477165313 shutdown_connections 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.228+0000 7efd44754640 1 -- 172.21.15.55:0/2477165313 wait complete. 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.229+0000 7efd44754640 10 monclient: build_initial_monmap 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.229+0000 7efd44754640 10 monclient: monmap: 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:33.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.229+0000 7efd44754640 1 librados: starting msgr at 2024-03-25T18:33:33.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.229+0000 7efd44754640 1 librados: starting objecter 2024-03-25T18:33:33.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.229+0000 7efd44754640 1 Processor -- start 2024-03-25T18:33:33.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.229+0000 7efd44754640 1 -- start start 2024-03-25T18:33:33.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.229+0000 7efd44754640 1 librados: setting wanted keys 2024-03-25T18:33:33.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.229+0000 7efd44754640 1 librados: calling monclient init 2024-03-25T18:33:33.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.229+0000 7efd44754640 10 monclient: init 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 10 monclient(hunting): picked mon.noname-c con 0x7efd3c0a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 10 monclient(hunting): picked mon.noname-b con 0x7efd3c13f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 10 monclient(hunting): picked mon.noname-a con 0x7efd3c1430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c136fa0 con 0x7efd3c1430c0 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c13b780 con 0x7efd3c13f710 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c1d7ba0 con 0x7efd3c0a98f0 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:33.234 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.230+0000 7efd44754640 10 monclient(hunting): authenticate will time out at 7617.488992s 2024-03-25T18:33:33.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.237+0000 7efd424f0640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7efd3c0a98f0 0x7efd3c136890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:44848/0 (socket says 172.21.15.55:44848) 2024-03-25T18:33:33.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.237+0000 7efd424f0640 1 -- 172.21.15.55:0/360035744 learned_addr learned my addr 172.21.15.55:0/360035744 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.239+0000 7efd32ffd640 10 client.?.objecter ms_handle_connect 0x7efd3c1430c0 2024-03-25T18:33:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.239+0000 7efd32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.240+0000 7efd32ffd640 10 client.?.objecter ms_handle_connect 0x7efd3c0a98f0 2024-03-25T18:33:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.240+0000 7efd32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.240+0000 7efd32ffd640 10 client.?.objecter ms_handle_connect 0x7efd3c13f710 2024-03-25T18:33:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.240+0000 7efd32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:33.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.251+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90370215 0 0) 0x7efd3c13b780 con 0x7efd3c13f710 2024-03-25T18:33:33.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.251+0000 7efd32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:33.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.251+0000 7efd32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:33.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.251+0000 7efd32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:33.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.251+0000 7efd32ffd640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:33:33.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.251+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd20004eb0 con 0x7efd3c13f710 2024-03-25T18:33:33.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.257+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443594078 0 0) 0x7efd3c136fa0 con 0x7efd3c1430c0 2024-03-25T18:33:33.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.257+0000 7efd32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:33.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.257+0000 7efd32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:33.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.257+0000 7efd32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:33.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.257+0000 7efd32ffd640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:33:33.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.257+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c13b780 con 0x7efd3c1430c0 2024-03-25T18:33:33.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.258+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348383006 0 0) 0x7efd3c1d7ba0 con 0x7efd3c0a98f0 2024-03-25T18:33:33.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.258+0000 7efd32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:33.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.258+0000 7efd32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:33.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.258+0000 7efd32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:33.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.258+0000 7efd32ffd640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:33:33.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.258+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c136fa0 con 0x7efd3c0a98f0 2024-03-25T18:33:33.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.265+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260188820 0 0) 0x7efd20004eb0 con 0x7efd3c13f710 2024-03-25T18:33:33.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.265+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd3c1d7ba0 con 0x7efd3c13f710 2024-03-25T18:33:33.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.265+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efd2c001ba0 con 0x7efd3c13f710 2024-03-25T18:33:33.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.265+0000 7efd32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:33.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.265+0000 7efd32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:33.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.265+0000 7efd32ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:33.267 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:33.267 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:33.268 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.268 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.268 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:33.268 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:33.268 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:33.268 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:33.268 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:33.268 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:33.268 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.265+0000 7efd32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:33.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.277+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 574089906 0 0) 0x7efd3c13b780 con 0x7efd3c1430c0 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.277+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd20004eb0 con 0x7efd3c1430c0 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.277+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efd34001b50 con 0x7efd3c1430c0 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.277+0000 7efd32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.277+0000 7efd32ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.277+0000 7efd32ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:33.280 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:33.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3492316881 0 0) 0x7efd3c1d7ba0 con 0x7efd3c13f710 2024-03-25T18:33:33.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 >> v1:172.21.15.55:6790/0 conn(0x7efd3c0a98f0 legacy=0x7efd3c136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:33.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 >> v1:172.21.15.55:6789/0 conn(0x7efd3c1430c0 legacy=0x7efd3c1e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:33.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:33.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 1 monclient: found mon.b 2024-03-25T18:33:33.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c137990 con 0x7efd3c13f710 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd44754640 5 monclient: authenticate success, global_id 4450 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd44754640 10 monclient: _renew_subs 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd44754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3c137c40 con 0x7efd3c13f710 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd44754640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd44754640 10 monclient: _renew_subs 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd44754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c139860 con 0x7efd3c13f710 2024-03-25T18:33:33.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.286+0000 7efd44754640 1 librados: init done 2024-03-25T18:33:33.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.287+0000 7efd44754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:33.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.287+0000 7efd44754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:33.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.287+0000 7efd44754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:33.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.287+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3c137640 con 0x7efd3c13f710 2024-03-25T18:33:33.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.583+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd2c004a90 con 0x7efd3c13f710 2024-03-25T18:33:33.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.583+0000 7efd32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:33.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.583+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efd2c004c00 con 0x7efd3c13f710 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.583+0000 7efd32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.583+0000 7efd32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.583+0000 7efd32ffd640 10 monclient: dump: 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:33.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.583+0000 7efd40ced640 4 set_mon_vals no callback set 2024-03-25T18:33:33.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.584+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7efd2c01abd0 con 0x7efd3c13f710 2024-03-25T18:33:33.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.584+0000 7efd424f0640 10 monclient: get_auth_request con 0x7efd2006df60 auth_method 0 2024-03-25T18:33:33.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.585+0000 7efd32ffd640 10 client.4450.objecter ms_handle_connect 0x7efd2006df60 2024-03-25T18:33:33.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.590+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7efd2c054770 con 0x7efd3c13f710 2024-03-25T18:33:33.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.590+0000 7efd32ffd640 10 client.4450.objecter ms_dispatch 0x7efd3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:33.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.590+0000 7efd32ffd640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:33.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.590+0000 7efd32ffd640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:33.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.590+0000 7efd32ffd640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T18:33:33.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.590+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd2c0500b0 con 0x7efd3c13f710 2024-03-25T18:33:33.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.590+0000 7efd32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:33.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.590+0000 7efd32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:33.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.838+0000 7efd44754640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:33.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.838+0000 7efd44754640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:33.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.838+0000 7efd44754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:33.840 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.838+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd3c0627b0 con 0x7efd3c13f710 2024-03-25T18:33:33.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.886+0000 7efd32ffd640 1 -- 172.21.15.55:0/360035744 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7efd2c0521b0 con 0x7efd3c13f710 2024-03-25T18:33:33.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.886+0000 7efd32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:33.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.886+0000 7efd32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:33.888 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:33:33.888 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:33.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 10 librados: watch_flush enter 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 10 librados: watch_flush exit 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 10 client.4450.objecter successfully canceled tick 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 >> v1:172.21.15.55:6800/286536064 conn(0x7efd2006df60 legacy=0x7efd20070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 10 monclient: shutdown 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 >> v1:172.21.15.133:6789/0 conn(0x7efd3c13f710 legacy=0x7efd3c1d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 shutdown_connections 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.888+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 >> 172.21.15.55:0/360035744 conn(0x7efd3c0b7ea0 msgr2=0x7efd3c14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.889+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 shutdown_connections 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.889+0000 7efd44754640 1 -- 172.21.15.55:0/360035744 wait complete. 2024-03-25T18:33:33.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:33.889+0000 7efd44754640 1 librados: shutdown 2024-03-25T18:33:33.902 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:35.839 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:05:46.009 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:35.843 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:05:46.013 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:35.875 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:05:46.045 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:35.896 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:46.033 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:35.905 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:46.042 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:35.906 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:05:46.076 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:35.916 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:05:46.086 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:35.946 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:05:46.083 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:35.954 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:05:46.124 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:40.710 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:40.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.821+0000 7f2e9e1ef640 10 monclient: get_monmap_and_config 2024-03-25T18:33:40.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.821+0000 7f2e9e1ef640 10 monclient: build_initial_monmap 2024-03-25T18:33:40.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.821+0000 7f2e9e1ef640 10 monclient: monmap: 2024-03-25T18:33:40.823 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:40.823 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:40.823 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:33:40.822780+0000 2024-03-25T18:33:40.823 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:33:40.822780+0000 2024-03-25T18:33:40.823 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:40.823 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:40.824 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:40.824 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:33:40.824 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:33:40.824 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:40.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.822+0000 7f2e9e1ef640 1 Processor -- start 2024-03-25T18:33:40.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.822+0000 7f2e9e1ef640 1 -- start start 2024-03-25T18:33:40.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.822+0000 7f2e9e1ef640 10 monclient: init 2024-03-25T18:33:40.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:40.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:40.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 10 monclient(hunting): picked mon.noname-a con 0x7f2e980a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:40.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9813f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:40.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 10 monclient(hunting): picked mon.noname-c con 0x7f2e981430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:40.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e98142520 con 0x7f2e980a98f0 2024-03-25T18:33:40.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980b28a0 con 0x7f2e981430c0 2024-03-25T18:33:40.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980b2a30 con 0x7f2e9813f710 2024-03-25T18:33:40.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:40.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.823+0000 7f2e9e1ef640 10 monclient(hunting): authenticate will time out at 7625.081113s 2024-03-25T18:33:40.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.836+0000 7f2e97fff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2e981430c0 0x7f2e9814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:44850/0 (socket says 172.21.15.55:44850) 2024-03-25T18:33:40.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.836+0000 7f2e97fff640 1 -- 172.21.15.55:0/631516769 learned_addr learned my addr 172.21.15.55:0/631516769 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:40.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.851+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773517376 0 0) 0x7f2e980b2a30 con 0x7f2e9813f710 2024-03-25T18:33:40.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.852+0000 7f2e967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.852+0000 7f2e967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.852+0000 7f2e967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.852+0000 7f2e967fc640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:33:40.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.852+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e80003520 con 0x7f2e9813f710 2024-03-25T18:33:40.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.853+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564927301 0 0) 0x7f2e980b28a0 con 0x7f2e981430c0 2024-03-25T18:33:40.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.853+0000 7f2e967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.853+0000 7f2e967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.853+0000 7f2e967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.853+0000 7f2e967fc640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:33:40.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.853+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e980b2a30 con 0x7f2e981430c0 2024-03-25T18:33:40.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.858+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573431621 0 0) 0x7f2e98142520 con 0x7f2e980a98f0 2024-03-25T18:33:40.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.858+0000 7f2e967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.858+0000 7f2e967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.858+0000 7f2e967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.858+0000 7f2e967fc640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:33:40.859 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.858+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e980b28a0 con 0x7f2e980a98f0 2024-03-25T18:33:40.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852297492 0 0) 0x7f2e80003520 con 0x7f2e9813f710 2024-03-25T18:33:40.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 >> v1:172.21.15.133:6789/0 conn(0x7f2e981430c0 legacy=0x7f2e9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.862 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 >> v1:172.21.15.55:6789/0 conn(0x7f2e980a98f0 legacy=0x7f2e980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 1 monclient: found mon.noname-b 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980b44c0 con 0x7f2e9813f710 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2e8c001690 con 0x7f2e9813f710 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e9e1ef640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e9e1ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:40.863 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e967fc640 10 monclient: dump: 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:40.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.861+0000 7f2e9e1ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:40.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.874+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e8c0017f0 con 0x7f2e9813f710 2024-03-25T18:33:40.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.874+0000 7f2e967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:40.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.874+0000 7f2e967fc640 1 -- 172.21.15.55:0/631516769 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2e8c001c00 con 0x7f2e9813f710 2024-03-25T18:33:40.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.874+0000 7f2e9e1ef640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.874+0000 7f2e9e1ef640 4 set_mon_vals no callback set 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.874+0000 7f2e967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.874+0000 7f2e967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.874+0000 7f2e967fc640 10 monclient: dump: 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:40.876 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:40.877 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:40.877 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:40.877 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:40.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.875+0000 7f2e9e1ef640 10 monclient: shutdown 2024-03-25T18:33:40.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.875+0000 7f2e9e1ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:40.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.875+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/631516769 >> v1:172.21.15.55:6790/0 conn(0x7f2e9813f710 legacy=0x7f2e981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.875+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/631516769 shutdown_connections 2024-03-25T18:33:40.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.875+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/631516769 >> 172.21.15.55:0/631516769 conn(0x7f2e980b7ea0 msgr2=0x7f2e980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:40.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.875+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/631516769 shutdown_connections 2024-03-25T18:33:40.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.875+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/631516769 wait complete. 2024-03-25T18:33:40.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.876+0000 7f2e9e1ef640 10 monclient: build_initial_monmap 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.876+0000 7f2e9e1ef640 10 monclient: monmap: 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.876+0000 7f2e9e1ef640 1 librados: starting msgr at 2024-03-25T18:33:40.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.876+0000 7f2e9e1ef640 1 librados: starting objecter 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 1 Processor -- start 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 1 -- start start 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 1 librados: setting wanted keys 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 1 librados: calling monclient init 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 10 monclient: init 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 10 monclient(hunting): picked mon.noname-c con 0x7f2e980a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 10 monclient(hunting): picked mon.noname-a con 0x7f2e981430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:40.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980ab8b0 con 0x7f2e981430c0 2024-03-25T18:33:40.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980abb90 con 0x7f2e9813f710 2024-03-25T18:33:40.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e98137870 con 0x7f2e980a98f0 2024-03-25T18:33:40.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:40.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.877+0000 7f2e9e1ef640 10 monclient(hunting): authenticate will time out at 7625.135196s 2024-03-25T18:33:40.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.884+0000 7f2e96ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f2e9813f710 0x7f2e981df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:44888/0 (socket says 172.21.15.55:44888) 2024-03-25T18:33:40.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.884+0000 7f2e96ffd640 1 -- 172.21.15.55:0/711829905 learned_addr learned my addr 172.21.15.55:0/711829905 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:40.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.888+0000 7f2e7ffff640 10 client.?.objecter ms_handle_connect 0x7f2e9813f710 2024-03-25T18:33:40.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.888+0000 7f2e7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:40.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.892+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309831467 0 0) 0x7f2e980abb90 con 0x7f2e9813f710 2024-03-25T18:33:40.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.892+0000 7f2e7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.892+0000 7f2e7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.892+0000 7f2e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.892+0000 7f2e7ffff640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:33:40.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.892+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e84004eb0 con 0x7f2e9813f710 2024-03-25T18:33:40.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.894+0000 7f2e7ffff640 10 client.?.objecter ms_handle_connect 0x7f2e980a98f0 2024-03-25T18:33:40.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.894+0000 7f2e7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:40.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.900+0000 7f2e7ffff640 10 client.?.objecter ms_handle_connect 0x7f2e981430c0 2024-03-25T18:33:40.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.900+0000 7f2e7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:40.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.912+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228520525 0 0) 0x7f2e84004eb0 con 0x7f2e9813f710 2024-03-25T18:33:40.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.912+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e980abb90 con 0x7f2e9813f710 2024-03-25T18:33:40.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.912+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2e8c001bc0 con 0x7f2e9813f710 2024-03-25T18:33:40.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.912+0000 7f2e7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:40.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.912+0000 7f2e7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:40.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.912+0000 7f2e7ffff640 10 monclient(hunting): dump: 2024-03-25T18:33:40.913 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:40.913 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:40.914 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:40.914 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:40.914 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:40.914 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:40.914 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:40.914 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:40.914 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:40.914 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:40.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.912+0000 7f2e7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:40.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.915+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899626667 0 0) 0x7f2e98137870 con 0x7f2e980a98f0 2024-03-25T18:33:40.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.915+0000 7f2e7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.915+0000 7f2e7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.915+0000 7f2e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.915+0000 7f2e7ffff640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:33:40.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.915+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e84004eb0 con 0x7f2e980a98f0 2024-03-25T18:33:40.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.922+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950514923 0 0) 0x7f2e980ab8b0 con 0x7f2e981430c0 2024-03-25T18:33:40.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.922+0000 7f2e7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:40.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.922+0000 7f2e7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:40.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.922+0000 7f2e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:40.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.922+0000 7f2e7ffff640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:33:40.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.922+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e98137870 con 0x7f2e981430c0 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1512756221 0 0) 0x7f2e980abb90 con 0x7f2e9813f710 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 >> v1:172.21.15.55:6790/0 conn(0x7f2e980a98f0 legacy=0x7f2e981350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 >> v1:172.21.15.55:6789/0 conn(0x7f2e981430c0 legacy=0x7f2e981e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 1 monclient: found mon.b 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e981389e0 con 0x7f2e9813f710 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 10 monclient: _finish_auth 0 2024-03-25T18:33:40.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e9e1ef640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e9e1ef640 10 monclient: _renew_subs 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e9e1ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e98136d90 con 0x7f2e9813f710 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e9e1ef640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e9e1ef640 10 monclient: _renew_subs 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e9e1ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e981384c0 con 0x7f2e9813f710 2024-03-25T18:33:40.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.930+0000 7f2e9e1ef640 1 librados: init done 2024-03-25T18:33:40.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.932+0000 7f2e9e1ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:40.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.932+0000 7f2e9e1ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:40.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.932+0000 7f2e9e1ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:40.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.932+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e980627b0 con 0x7f2e9813f710 2024-03-25T18:33:40.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.933+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e8c0011f0 con 0x7f2e9813f710 2024-03-25T18:33:40.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.933+0000 7f2e7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:40.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:40.933+0000 7f2e95ffb640 4 set_mon_vals no callback set 2024-03-25T18:33:41.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.212+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2e8c003b00 con 0x7f2e9813f710 2024-03-25T18:33:41.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.212+0000 7f2e7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:41.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.212+0000 7f2e7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.212+0000 7f2e7ffff640 10 monclient: dump: 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:41.214 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:41.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.213+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f2e8c003d20 con 0x7f2e9813f710 2024-03-25T18:33:41.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.214+0000 7f2e977fe640 10 monclient: get_auth_request con 0x7f2e8406de10 auth_method 0 2024-03-25T18:33:41.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.214+0000 7f2e7ffff640 10 client.4456.objecter ms_handle_connect 0x7f2e8406de10 2024-03-25T18:33:41.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.216+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f2e8c0549a0 con 0x7f2e9813f710 2024-03-25T18:33:41.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.216+0000 7f2e7ffff640 10 client.4456.objecter ms_dispatch 0x7f2e98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:41.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.216+0000 7f2e7ffff640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:41.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.216+0000 7f2e7ffff640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:41.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.216+0000 7f2e7ffff640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T18:33:41.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.219+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e8c0502e0 con 0x7f2e9813f710 2024-03-25T18:33:41.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.219+0000 7f2e7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:41.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.219+0000 7f2e7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:41.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.471+0000 7f2e9e1ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:41.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.471+0000 7f2e9e1ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:41.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.471+0000 7f2e9e1ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:41.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.471+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e98138b70 con 0x7f2e9813f710 2024-03-25T18:33:41.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.508+0000 7f2e7ffff640 1 -- 172.21.15.55:0/711829905 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f2e8c0523e0 con 0x7f2e9813f710 2024-03-25T18:33:41.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.508+0000 7f2e7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:41.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.508+0000 7f2e7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:41.510 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:33:41.510 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:41.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 10 librados: watch_flush enter 2024-03-25T18:33:41.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 10 librados: watch_flush exit 2024-03-25T18:33:41.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T18:33:41.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 10 client.4456.objecter successfully canceled tick 2024-03-25T18:33:41.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 >> v1:172.21.15.55:6800/286536064 conn(0x7f2e8406de10 legacy=0x7f2e840702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:41.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 10 monclient: shutdown 2024-03-25T18:33:41.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:41.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 >> v1:172.21.15.133:6789/0 conn(0x7f2e9813f710 legacy=0x7f2e981df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:41.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 shutdown_connections 2024-03-25T18:33:41.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.510+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 >> 172.21.15.55:0/711829905 conn(0x7f2e980b7ea0 msgr2=0x7f2e98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:41.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.511+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 shutdown_connections 2024-03-25T18:33:41.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.511+0000 7f2e9e1ef640 1 -- 172.21.15.55:0/711829905 wait complete. 2024-03-25T18:33:41.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:41.511+0000 7f2e9e1ef640 1 librados: shutdown 2024-03-25T18:33:41.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:48.329 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:48.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.440+0000 7fb7b256b640 10 monclient: get_monmap_and_config 2024-03-25T18:33:48.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.440+0000 7fb7b256b640 10 monclient: build_initial_monmap 2024-03-25T18:33:48.441 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.440+0000 7fb7b256b640 10 monclient: monmap: 2024-03-25T18:33:48.441 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:48.441 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:48.441 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:33:48.441280+0000 2024-03-25T18:33:48.441 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:33:48.441280+0000 2024-03-25T18:33:48.442 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:48.442 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:48.442 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:48.442 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:33:48.442 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:33:48.442 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:48.443 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.441+0000 7fb7b256b640 1 Processor -- start 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.441+0000 7fb7b256b640 1 -- start start 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.441+0000 7fb7b256b640 10 monclient: init 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 10 monclient(hunting): picked mon.noname-b con 0x7fb7ac0b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 10 monclient(hunting): picked mon.noname-c con 0x7fb7ac0b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 10 monclient(hunting): picked mon.noname-a con 0x7fb7ac14be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac0af940 con 0x7fb7ac14be90 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac0b0a70 con 0x7fb7ac0b1730 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac0b0d50 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:48.444 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.442+0000 7fb7b256b640 10 monclient(hunting): authenticate will time out at 7632.699668s 2024-03-25T18:33:48.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.451+0000 7fb7ab7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fb7ac0b1730 0x7fb7ac0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:49596/0 (socket says 172.21.15.55:49596) 2024-03-25T18:33:48.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.451+0000 7fb7ab7fe640 1 -- 172.21.15.55:0/1996505815 learned_addr learned my addr 172.21.15.55:0/1996505815 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:48.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.463+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713242287 0 0) 0x7fb7ac0b0a70 con 0x7fb7ac0b1730 2024-03-25T18:33:48.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.463+0000 7fb7aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:48.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.463+0000 7fb7aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:48.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.463+0000 7fb7aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:48.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.463+0000 7fb7aaffd640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:33:48.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.463+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb794003520 con 0x7fb7ac0b1730 2024-03-25T18:33:48.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.472+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818105106 0 0) 0x7fb7ac0b0d50 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.473 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.472+0000 7fb7aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:48.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.472+0000 7fb7aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:48.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.472+0000 7fb7aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:48.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.472+0000 7fb7aaffd640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:33:48.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.472+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ac0b0a70 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1894099210 0 0) 0x7fb794003520 con 0x7fb7ac0b1730 2024-03-25T18:33:48.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 >> v1:172.21.15.55:6790/0 conn(0x7fb7ac0b4d60 legacy=0x7fb7ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:48.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 >> v1:172.21.15.55:6789/0 conn(0x7fb7ac14be90 legacy=0x7fb7ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:48.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:48.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 1 monclient: found mon.noname-c 2024-03-25T18:33:48.481 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac0b1030 con 0x7fb7ac0b1730 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb7a0001690 con 0x7fb7ac0b1730 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7aaffd640 10 monclient: dump: 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:48.482 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:48.483 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:48.483 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:48.483 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:48.483 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:48.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7b256b640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:33:48.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.480+0000 7fb7b256b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:48.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.497+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7a00017f0 con 0x7fb7ac0b1730 2024-03-25T18:33:48.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.497+0000 7fb7aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.497+0000 7fb7aaffd640 1 -- 172.21.15.55:0/1996505815 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb7a0001c00 con 0x7fb7ac0b1730 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.497+0000 7fb7b256b640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.497+0000 7fb7b256b640 4 set_mon_vals no callback set 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.497+0000 7fb7aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.498+0000 7fb7aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.498+0000 7fb7aaffd640 10 monclient: dump: 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:48.499 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:48.500 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:48.500 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:48.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.498+0000 7fb7b256b640 10 monclient: shutdown 2024-03-25T18:33:48.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.498+0000 7fb7b256b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:48.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.498+0000 7fb7b256b640 1 -- 172.21.15.55:0/1996505815 >> v1:172.21.15.133:6789/0 conn(0x7fb7ac0b1730 legacy=0x7fb7ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:48.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.498+0000 7fb7b256b640 1 -- 172.21.15.55:0/1996505815 shutdown_connections 2024-03-25T18:33:48.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.498+0000 7fb7b256b640 1 -- 172.21.15.55:0/1996505815 >> 172.21.15.55:0/1996505815 conn(0x7fb7ac0617a0 msgr2=0x7fb7ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:48.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.498+0000 7fb7b256b640 1 -- 172.21.15.55:0/1996505815 shutdown_connections 2024-03-25T18:33:48.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.498+0000 7fb7b256b640 1 -- 172.21.15.55:0/1996505815 wait complete. 2024-03-25T18:33:48.501 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.499+0000 7fb7b256b640 10 monclient: build_initial_monmap 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.499+0000 7fb7b256b640 10 monclient: monmap: 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.500+0000 7fb7b256b640 1 librados: starting msgr at 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.500+0000 7fb7b256b640 1 librados: starting objecter 2024-03-25T18:33:48.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.500+0000 7fb7b256b640 1 Processor -- start 2024-03-25T18:33:48.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.500+0000 7fb7b256b640 1 -- start start 2024-03-25T18:33:48.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.500+0000 7fb7b256b640 1 librados: setting wanted keys 2024-03-25T18:33:48.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.500+0000 7fb7b256b640 1 librados: calling monclient init 2024-03-25T18:33:48.503 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.500+0000 7fb7b256b640 10 monclient: init 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 10 monclient(hunting): picked mon.noname-a con 0x7fb7ac0b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 10 monclient(hunting): picked mon.noname-b con 0x7fb7ac0b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 10 monclient(hunting): picked mon.noname-c con 0x7fb7ac14be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac1d7dd0 con 0x7fb7ac0b1730 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac1d7980 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac1d7b10 con 0x7fb7ac14be90 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:48.504 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.501+0000 7fb7b256b640 10 monclient(hunting): authenticate will time out at 7632.758949s 2024-03-25T18:33:48.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.513+0000 7fb7abfff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fb7ac0b1730 0x7fb7ac13ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:35900/0 (socket says 172.21.15.55:35900) 2024-03-25T18:33:48.514 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.513+0000 7fb7abfff640 1 -- 172.21.15.55:0/731253479 learned_addr learned my addr 172.21.15.55:0/731253479 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:48.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.514+0000 7fb7a8ff9640 10 client.?.objecter ms_handle_connect 0x7fb7ac0b1730 2024-03-25T18:33:48.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.515+0000 7fb7a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:48.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.516+0000 7fb7a8ff9640 10 client.?.objecter ms_handle_connect 0x7fb7ac14be90 2024-03-25T18:33:48.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.516+0000 7fb7a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:48.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.516+0000 7fb7a8ff9640 10 client.?.objecter ms_handle_connect 0x7fb7ac0b4d60 2024-03-25T18:33:48.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.516+0000 7fb7a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:48.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.520+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278653326 0 0) 0x7fb7ac1d7980 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.520+0000 7fb7a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:48.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.520+0000 7fb7a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:48.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.520+0000 7fb7a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:48.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.520+0000 7fb7a8ff9640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:33:48.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.520+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb798004eb0 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.524+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790907489 0 0) 0x7fb7ac1d7dd0 con 0x7fb7ac0b1730 2024-03-25T18:33:48.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.524+0000 7fb7a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:48.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.524+0000 7fb7a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:48.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.524+0000 7fb7a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:48.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.524+0000 7fb7a8ff9640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:33:48.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.524+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ac1d7980 con 0x7fb7ac0b1730 2024-03-25T18:33:48.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.525+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076493028 0 0) 0x7fb7ac1d7b10 con 0x7fb7ac14be90 2024-03-25T18:33:48.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.525+0000 7fb7a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:48.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.525+0000 7fb7a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:48.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.525+0000 7fb7a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:48.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.525+0000 7fb7a8ff9640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:33:48.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.525+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ac1d7dd0 con 0x7fb7ac14be90 2024-03-25T18:33:48.529 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.529+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2636284750 0 0) 0x7fb798004eb0 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.529+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7ac1d7b10 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.529+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb7a0001b00 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.529+0000 7fb7a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.529+0000 7fb7a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.529+0000 7fb7a8ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:48.530 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:48.531 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:48.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.529+0000 7fb7a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:48.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.536+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2810785854 0 0) 0x7fb7ac1d7dd0 con 0x7fb7ac14be90 2024-03-25T18:33:48.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.536+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb798004eb0 con 0x7fb7ac14be90 2024-03-25T18:33:48.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.536+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb7a4001ad0 con 0x7fb7ac14be90 2024-03-25T18:33:48.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.536+0000 7fb7a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:48.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.536+0000 7fb7a8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:48.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.536+0000 7fb7a8ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:48.537 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:48.538 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:48.538 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.538 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.538 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:48.538 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:48.538 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:48.538 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:48.538 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:48.538 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 296779553 0 0) 0x7fb7ac1d7b10 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 >> v1:172.21.15.55:6790/0 conn(0x7fb7ac14be90 legacy=0x7fb7ac1d8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 >> v1:172.21.15.55:6789/0 conn(0x7fb7ac0b1730 legacy=0x7fb7ac13ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 1 monclient: found mon.b 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac139290 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:48.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:48.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7b256b640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:33:48.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7b256b640 10 monclient: _renew_subs 2024-03-25T18:33:48.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7b256b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:48.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.541+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ac139540 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.542+0000 7fb7b256b640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:48.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.542+0000 7fb7b256b640 10 monclient: _renew_subs 2024-03-25T18:33:48.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.542+0000 7fb7b256b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:48.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.542+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac1e3620 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.542+0000 7fb7b256b640 1 librados: init done 2024-03-25T18:33:48.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.543+0000 7fb7b256b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:48.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.543+0000 7fb7b256b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:48.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.543+0000 7fb7b256b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:48.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.543+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac0627b0 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.548+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7a0004a90 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.548+0000 7fb7a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:48.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.548+0000 7fb7aa7fc640 4 set_mon_vals no callback set 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.550+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb7a0004da0 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.550+0000 7fb7a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.550+0000 7fb7a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.550+0000 7fb7a8ff9640 10 monclient: dump: 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:48.551 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:48.552 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:48.823 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.823+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fb7a001ad70 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.824 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.823+0000 7fb7abfff640 10 monclient: get_auth_request con 0x7fb79806df10 auth_method 0 2024-03-25T18:33:48.824 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.823+0000 7fb7a8ff9640 10 client.4462.objecter ms_handle_connect 0x7fb79806df10 2024-03-25T18:33:48.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.831+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fb7a0054880 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.831+0000 7fb7a8ff9640 10 client.4462.objecter ms_dispatch 0x7fb7ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:48.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.831+0000 7fb7a8ff9640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:48.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.831+0000 7fb7a8ff9640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:48.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.831+0000 7fb7a8ff9640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T18:33:48.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.832+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7a00501c0 con 0x7fb7ac0b4d60 2024-03-25T18:33:48.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.832+0000 7fb7a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:48.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:48.832+0000 7fb7a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:49.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.086+0000 7fb7b256b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:49.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.086+0000 7fb7b256b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:49.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.086+0000 7fb7b256b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:33:49.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.086+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7ac069f40 con 0x7fb7ac0b4d60 2024-03-25T18:33:49.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.111+0000 7fb7a8ff9640 1 -- 172.21.15.55:0/731253479 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fb7a00522c0 con 0x7fb7ac0b4d60 2024-03-25T18:33:49.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.111+0000 7fb7a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:49.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.111+0000 7fb7a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:49.112 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:33:49.112 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.113+0000 7fb7b256b640 10 librados: watch_flush enter 2024-03-25T18:33:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.113+0000 7fb7b256b640 10 librados: watch_flush exit 2024-03-25T18:33:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.113+0000 7fb7b256b640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T18:33:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.113+0000 7fb7b256b640 10 client.4462.objecter successfully canceled tick 2024-03-25T18:33:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.114+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 >> v1:172.21.15.55:6800/286536064 conn(0x7fb79806df10 legacy=0x7fb7980703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.114+0000 7fb7b256b640 10 monclient: shutdown 2024-03-25T18:33:49.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.114+0000 7fb7b256b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:49.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.114+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 >> v1:172.21.15.133:6789/0 conn(0x7fb7ac0b4d60 legacy=0x7fb7ac142780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:49.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.114+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 shutdown_connections 2024-03-25T18:33:49.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.114+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 >> 172.21.15.55:0/731253479 conn(0x7fb7ac0617a0 msgr2=0x7fb7ac137410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:49.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.114+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 shutdown_connections 2024-03-25T18:33:49.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.114+0000 7fb7b256b640 1 -- 172.21.15.55:0/731253479 wait complete. 2024-03-25T18:33:49.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:49.114+0000 7fb7b256b640 1 librados: shutdown 2024-03-25T18:33:49.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:55.867 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:06:06.037 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:55.873 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:06:06.043 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:55.875 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:06:06.045 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:55.912 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:06:06.082 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:55.922 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:06.059 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:55.933 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:55.934 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:06.071 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:55.940 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:06:06.110 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:55.969 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:06:06.137 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:55.969 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:06.104 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:56.044 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.043+0000 7fdccced9640 10 monclient: get_monmap_and_config 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.043+0000 7fdccced9640 10 monclient: build_initial_monmap 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.043+0000 7fdccced9640 10 monclient: monmap: 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:33:56.044643+0000 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:33:56.044643+0000 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:33:56.045 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:56.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.044+0000 7fdccced9640 1 Processor -- start 2024-03-25T18:33:56.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.044+0000 7fdccced9640 1 -- start start 2024-03-25T18:33:56.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.044+0000 7fdccced9640 10 monclient: init 2024-03-25T18:33:56.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:56.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:56.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc806adf0 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:56.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc80aee10 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:56.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc8140860 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:56.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80af690 con 0x7fdcc80aee10 2024-03-25T18:33:56.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80b0c70 con 0x7fdcc8140860 2024-03-25T18:33:56.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80b0f50 con 0x7fdcc806adf0 2024-03-25T18:33:56.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:56.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.045+0000 7fdccced9640 10 monclient(hunting): authenticate will time out at 7640.302849s 2024-03-25T18:33:56.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.072+0000 7fdcc659c640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fdcc806adf0 0x7fdcc813d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:49262/0 (socket says 172.21.15.55:49262) 2024-03-25T18:33:56.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.073+0000 7fdcc659c640 1 -- 172.21.15.55:0/2617931430 learned_addr learned my addr 172.21.15.55:0/2617931430 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:56.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.098+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47564251 0 0) 0x7fdcc80b0c70 con 0x7fdcc8140860 2024-03-25T18:33:56.099 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.098+0000 7fdcc5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:56.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.098+0000 7fdcc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:56.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.098+0000 7fdcc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:56.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.098+0000 7fdcc5d9b640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:33:56.100 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.098+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb4003520 con 0x7fdcc8140860 2024-03-25T18:33:56.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.118+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457133261 0 0) 0x7fdcc80af690 con 0x7fdcc80aee10 2024-03-25T18:33:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.118+0000 7fdcc5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.118+0000 7fdcc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.118+0000 7fdcc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.118+0000 7fdcc5d9b640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:33:56.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.118+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc80b0c70 con 0x7fdcc80aee10 2024-03-25T18:33:56.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3183823909 0 0) 0x7fdcb4003520 con 0x7fdcc8140860 2024-03-25T18:33:56.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 >> v1:172.21.15.55:6790/0 conn(0x7fdcc806adf0 legacy=0x7fdcc813d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:56.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 >> v1:172.21.15.55:6789/0 conn(0x7fdcc80aee10 legacy=0x7fdcc8140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:56.128 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 1 monclient: found mon.noname-c 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc813dc40 con 0x7fdcc8140860 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdcc0001a10 con 0x7fdcc8140860 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdcc5d9b640 10 monclient: dump: 2024-03-25T18:33:56.129 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdccced9640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:33:56.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.127+0000 7fdccced9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:56.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.140+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcc0001b70 con 0x7fdcc8140860 2024-03-25T18:33:56.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.140+0000 7fdcc5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:56.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.140+0000 7fdcc5d9b640 1 -- 172.21.15.55:0/2617931430 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdcc0001f80 con 0x7fdcc8140860 2024-03-25T18:33:56.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.140+0000 7fdccced9640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:56.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.140+0000 7fdccced9640 4 set_mon_vals no callback set 2024-03-25T18:33:56.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.140+0000 7fdcc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:56.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.140+0000 7fdcc5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:56.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.140+0000 7fdcc5d9b640 10 monclient: dump: 2024-03-25T18:33:56.141 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:56.142 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:56.142 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.142 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.142 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:56.142 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:56.142 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:56.142 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:56.142 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:56.142 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:56.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.141+0000 7fdccced9640 10 monclient: shutdown 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.141+0000 7fdccced9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.141+0000 7fdccced9640 1 -- 172.21.15.55:0/2617931430 >> v1:172.21.15.133:6789/0 conn(0x7fdcc8140860 legacy=0x7fdcc814b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.141+0000 7fdccced9640 1 -- 172.21.15.55:0/2617931430 shutdown_connections 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.141+0000 7fdccced9640 1 -- 172.21.15.55:0/2617931430 >> 172.21.15.55:0/2617931430 conn(0x7fdcc80b4a80 msgr2=0x7fdcc80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.141+0000 7fdccced9640 1 -- 172.21.15.55:0/2617931430 shutdown_connections 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.141+0000 7fdccced9640 1 -- 172.21.15.55:0/2617931430 wait complete. 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.142+0000 7fdccced9640 10 monclient: build_initial_monmap 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.142+0000 7fdccced9640 10 monclient: monmap: 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:56.145 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.142+0000 7fdccced9640 1 librados: starting msgr at 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.142+0000 7fdccced9640 1 librados: starting objecter 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.142+0000 7fdccced9640 1 Processor -- start 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.142+0000 7fdccced9640 1 -- start start 2024-03-25T18:33:56.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 1 librados: setting wanted keys 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 1 librados: calling monclient init 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 10 monclient: init 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc806adf0 addr v1:172.21.15.55:6789/0 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc80aee10 addr v1:172.21.15.133:6789/0 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc8131330 addr v1:172.21.15.55:6790/0 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc8127af0 con 0x7fdcc806adf0 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc813cb50 con 0x7fdcc80aee10 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc813afc0 con 0x7fdcc8131330 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:56.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.143+0000 7fdccced9640 10 monclient(hunting): authenticate will time out at 7640.400913s 2024-03-25T18:33:56.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.156+0000 7fdcbffff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fdcc80aee10 0x7fdcc8137d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:59782/0 (socket says 172.21.15.55:59782) 2024-03-25T18:33:56.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.156+0000 7fdcbffff640 1 -- 172.21.15.55:0/249155658 learned_addr learned my addr 172.21.15.55:0/249155658 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:33:56.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.161+0000 7fdcbeffd640 10 client.?.objecter ms_handle_connect 0x7fdcc806adf0 2024-03-25T18:33:56.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.161+0000 7fdcbeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:56.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.163+0000 7fdcbeffd640 10 client.?.objecter ms_handle_connect 0x7fdcc80aee10 2024-03-25T18:33:56.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.163+0000 7fdcbeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:56.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.174+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785405920 0 0) 0x7fdcc813cb50 con 0x7fdcc80aee10 2024-03-25T18:33:56.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.174+0000 7fdcbeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:56.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.174+0000 7fdcbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:56.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.174+0000 7fdcbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:56.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.174+0000 7fdcbeffd640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:33:56.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.174+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca8004eb0 con 0x7fdcc80aee10 2024-03-25T18:33:56.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.181+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11052099 0 0) 0x7fdcc8127af0 con 0x7fdcc806adf0 2024-03-25T18:33:56.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.181+0000 7fdcbeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:56.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.181+0000 7fdcbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:56.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.181+0000 7fdcbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:56.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.181+0000 7fdcbeffd640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:33:56.182 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.181+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc813cb50 con 0x7fdcc806adf0 2024-03-25T18:33:56.193 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.192+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2371187785 0 0) 0x7fdcc813cb50 con 0x7fdcc806adf0 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.193+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc8127af0 con 0x7fdcc806adf0 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.193+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdcb8001510 con 0x7fdcc806adf0 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.193+0000 7fdcbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.193+0000 7fdcbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.193+0000 7fdcbeffd640 10 monclient(hunting): dump: 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.194 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:56.195 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:56.195 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:56.195 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:56.195 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:56.195 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.193+0000 7fdcbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.194+0000 7fdcbeffd640 10 client.?.objecter ms_handle_connect 0x7fdcc8131330 2024-03-25T18:33:56.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.194+0000 7fdcbeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:56.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905066979 0 0) 0x7fdcc813afc0 con 0x7fdcc8131330 2024-03-25T18:33:56.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:56.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:56.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:56.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:33:56.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc813cb50 con 0x7fdcc8131330 2024-03-25T18:33:56.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 206483790 0 0) 0x7fdca8004eb0 con 0x7fdcc80aee10 2024-03-25T18:33:56.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc813afc0 con 0x7fdcc80aee10 2024-03-25T18:33:56.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdcb0001a10 con 0x7fdcc80aee10 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.201+0000 7fdcbeffd640 10 monclient(hunting): dump: 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:56.204 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:56.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1987681407 0 0) 0x7fdcc8127af0 con 0x7fdcc806adf0 2024-03-25T18:33:56.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 >> v1:172.21.15.55:6790/0 conn(0x7fdcc8131330 legacy=0x7fdcc8138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:56.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 >> v1:172.21.15.133:6789/0 conn(0x7fdcc80aee10 legacy=0x7fdcc8137d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:56.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:56.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 1 monclient: found mon.a 2024-03-25T18:33:56.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:56.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc813c040 con 0x7fdcc806adf0 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdcbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdccced9640 5 monclient: authenticate success, global_id 4497 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdccced9640 10 monclient: _renew_subs 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdccced9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.202+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc813c230 con 0x7fdcc806adf0 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.203+0000 7fdccced9640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.203+0000 7fdccced9640 10 monclient: _renew_subs 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.203+0000 7fdccced9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:56.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.203+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc8138d80 con 0x7fdcc806adf0 2024-03-25T18:33:56.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.203+0000 7fdccced9640 1 librados: init done 2024-03-25T18:33:56.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.204+0000 7fdccced9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:56.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.204+0000 7fdccced9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:56.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.204+0000 7fdccced9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:56.208 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.204+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc8139080 con 0x7fdcc806adf0 2024-03-25T18:33:56.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.516+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcb8001a40 con 0x7fdcc806adf0 2024-03-25T18:33:56.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.516+0000 7fdcbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:56.517 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.516+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fdcb8004b10 con 0x7fdcc806adf0 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.516+0000 7fdcbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.516+0000 7fdcbeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.516+0000 7fdcbeffd640 10 monclient: dump: 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:33:56.518 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:33:56.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.516+0000 7fdcc559a640 4 set_mon_vals no callback set 2024-03-25T18:33:56.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.517+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fdcb801aae0 con 0x7fdcc806adf0 2024-03-25T18:33:56.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.518+0000 7fdcbffff640 10 monclient: get_auth_request con 0x7fdca806df20 auth_method 0 2024-03-25T18:33:56.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.518+0000 7fdcbeffd640 10 client.4497.objecter ms_handle_connect 0x7fdca806df20 2024-03-25T18:33:56.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.524+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fdcb8054750 con 0x7fdcc806adf0 2024-03-25T18:33:56.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.524+0000 7fdcbeffd640 10 client.4497.objecter ms_dispatch 0x7fdcc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:56.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.524+0000 7fdcbeffd640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:56.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.524+0000 7fdcbeffd640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:56.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.524+0000 7fdcbeffd640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T18:33:56.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.524+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcb8050090 con 0x7fdcc806adf0 2024-03-25T18:33:56.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.524+0000 7fdcbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:56.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.524+0000 7fdcbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:56.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.745+0000 7fdccced9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:56.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.746+0000 7fdccced9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:56.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.746+0000 7fdccced9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:33:56.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.746+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcc813cce0 con 0x7fdcc806adf0 2024-03-25T18:33:56.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.770+0000 7fdcbeffd640 1 -- 172.21.15.55:0/249155658 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fdcb8052190 con 0x7fdcc806adf0 2024-03-25T18:33:56.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.770+0000 7fdcbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:56.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.770+0000 7fdcbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:56.772 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:33:56.772 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:56.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.772+0000 7fdccced9640 10 librados: watch_flush enter 2024-03-25T18:33:56.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 10 librados: watch_flush exit 2024-03-25T18:33:56.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T18:33:56.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 10 client.4497.objecter successfully canceled tick 2024-03-25T18:33:56.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 >> v1:172.21.15.55:6800/286536064 conn(0x7fdca806df20 legacy=0x7fdca80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:56.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 10 monclient: shutdown 2024-03-25T18:33:56.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:56.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 >> v1:172.21.15.55:6789/0 conn(0x7fdcc806adf0 legacy=0x7fdcc8125f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:56.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 shutdown_connections 2024-03-25T18:33:56.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 >> 172.21.15.55:0/249155658 conn(0x7fdcc80b4a80 msgr2=0x7fdcc8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:56.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 shutdown_connections 2024-03-25T18:33:56.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 1 -- 172.21.15.55:0/249155658 wait complete. 2024-03-25T18:33:56.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:33:56.773+0000 7fdccced9640 1 librados: shutdown 2024-03-25T18:33:56.789 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:03.595 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.711+0000 7f636993d640 10 monclient: get_monmap_and_config 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.711+0000 7f636993d640 10 monclient: build_initial_monmap 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.711+0000 7f636993d640 10 monclient: monmap: 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:34:03.712076+0000 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:34:03.712076+0000 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:34:03.712 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:34:03.713 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.711+0000 7f636993d640 1 Processor -- start 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.712+0000 7f636993d640 1 -- start start 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.712+0000 7f636993d640 10 monclient: init 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.712+0000 7f636993d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.712+0000 7f636993d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.712+0000 7f636993d640 10 monclient(hunting): picked mon.noname-a con 0x7f63640a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.712+0000 7f636993d640 10 monclient(hunting): picked mon.noname-c con 0x7f636413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.712+0000 7f636993d640 10 monclient(hunting): picked mon.noname-b con 0x7f63641430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.713+0000 7f636993d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6364142520 con 0x7f63640a98f0 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.713+0000 7f636993d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63640b28a0 con 0x7f636413f710 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.713+0000 7f636993d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63640b2a30 con 0x7f63641430c0 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.713+0000 7f636993d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:03.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.713+0000 7f636993d640 10 monclient(hunting): authenticate will time out at 7647.970398s 2024-03-25T18:34:03.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.724+0000 7f63627fc640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f636413f710 0x7f63641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:50712/0 (socket says 172.21.15.55:50712) 2024-03-25T18:34:03.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.724+0000 7f63627fc640 1 -- 172.21.15.55:0/63593895 learned_addr learned my addr 172.21.15.55:0/63593895 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:03.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.738+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885024789 0 0) 0x7f63640b2a30 con 0x7f63641430c0 2024-03-25T18:34:03.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.738+0000 7f6361ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.738+0000 7f6361ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.738+0000 7f6361ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.738+0000 7f6361ffb640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:34:03.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.738+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c003520 con 0x7f63641430c0 2024-03-25T18:34:03.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.743+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525078926 0 0) 0x7f63640b28a0 con 0x7f636413f710 2024-03-25T18:34:03.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.743+0000 7f6361ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.743+0000 7f6361ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.743+0000 7f6361ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.743+0000 7f6361ffb640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:34:03.744 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.743+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63640b2a30 con 0x7f636413f710 2024-03-25T18:34:03.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1270314431 0 0) 0x7f634c003520 con 0x7f63641430c0 2024-03-25T18:34:03.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 >> v1:172.21.15.133:6789/0 conn(0x7f636413f710 legacy=0x7f63641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 >> v1:172.21.15.55:6789/0 conn(0x7f63640a98f0 legacy=0x7f63640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 1 monclient: found mon.noname-b 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63640b44c0 con 0x7f63641430c0 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 10 monclient: _finish_auth 0 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f635c001a10 con 0x7f63641430c0 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f6361ffb640 10 monclient: dump: 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:03.751 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:03.752 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:03.752 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:03.752 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:03.752 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:03.752 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:03.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f636993d640 5 monclient: authenticate success, global_id 4484 2024-03-25T18:34:03.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.749+0000 7f636993d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:03.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.759+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f635c001b70 con 0x7f63641430c0 2024-03-25T18:34:03.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.759+0000 7f6361ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:03.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.759+0000 7f6361ffb640 1 -- 172.21.15.55:0/63593895 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f635c001f80 con 0x7f63641430c0 2024-03-25T18:34:03.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.759+0000 7f6361ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:03.760 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.759+0000 7f6361ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.759+0000 7f6361ffb640 10 monclient: dump: 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.759+0000 7f636993d640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:03.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.759+0000 7f636993d640 4 set_mon_vals no callback set 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.760+0000 7f636993d640 10 monclient: shutdown 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.760+0000 7f636993d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.760+0000 7f636993d640 1 -- 172.21.15.55:0/63593895 >> v1:172.21.15.55:6790/0 conn(0x7f63641430c0 legacy=0x7f636414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.760+0000 7f636993d640 1 -- 172.21.15.55:0/63593895 shutdown_connections 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.760+0000 7f636993d640 1 -- 172.21.15.55:0/63593895 >> 172.21.15.55:0/63593895 conn(0x7f63640b7ea0 msgr2=0x7f63640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.760+0000 7f636993d640 1 -- 172.21.15.55:0/63593895 shutdown_connections 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.760+0000 7f636993d640 1 -- 172.21.15.55:0/63593895 wait complete. 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.761+0000 7f636993d640 10 monclient: build_initial_monmap 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.761+0000 7f636993d640 10 monclient: monmap: 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:34:03.763 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.761+0000 7f636993d640 1 librados: starting msgr at 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.761+0000 7f636993d640 1 librados: starting objecter 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 1 Processor -- start 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 1 -- start start 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 1 librados: setting wanted keys 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 1 librados: calling monclient init 2024-03-25T18:34:03.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 10 monclient: init 2024-03-25T18:34:03.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:03.765 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:03.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 10 monclient(hunting): picked mon.noname-a con 0x7f63640a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:03.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 10 monclient(hunting): picked mon.noname-b con 0x7f636413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:03.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 10 monclient(hunting): picked mon.noname-c con 0x7f63641430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:03.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6364136ad0 con 0x7f63640a98f0 2024-03-25T18:34:03.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63640b7520 con 0x7f636413f710 2024-03-25T18:34:03.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6364135810 con 0x7f63641430c0 2024-03-25T18:34:03.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.762+0000 7f636993d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:03.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.763+0000 7f636993d640 10 monclient(hunting): authenticate will time out at 7648.020372s 2024-03-25T18:34:03.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.775+0000 7f63627fc640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f636413f710 0x7f63640b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:50748/0 (socket says 172.21.15.55:50748) 2024-03-25T18:34:03.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.775+0000 7f63627fc640 1 -- 172.21.15.55:0/1643913316 learned_addr learned my addr 172.21.15.55:0/1643913316 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:03.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.776+0000 7f634bfff640 10 client.?.objecter ms_handle_connect 0x7f636413f710 2024-03-25T18:34:03.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.776+0000 7f634bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:03.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.776+0000 7f634bfff640 10 client.?.objecter ms_handle_connect 0x7f63640a98f0 2024-03-25T18:34:03.777 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.776+0000 7f634bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:03.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.784+0000 7f634bfff640 10 client.?.objecter ms_handle_connect 0x7f63641430c0 2024-03-25T18:34:03.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.784+0000 7f634bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:03.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.784+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8702963 0 0) 0x7f6364136ad0 con 0x7f63640a98f0 2024-03-25T18:34:03.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.784+0000 7f634bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.784+0000 7f634bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.784+0000 7f634bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.784+0000 7f634bfff640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:34:03.785 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.784+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6350004eb0 con 0x7f63640a98f0 2024-03-25T18:34:03.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.796+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610163076 0 0) 0x7f63640b7520 con 0x7f636413f710 2024-03-25T18:34:03.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.796+0000 7f634bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:03.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.796+0000 7f634bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:03.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.796+0000 7f634bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:03.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.796+0000 7f634bfff640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:34:03.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.796+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6364136ad0 con 0x7f636413f710 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.798+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1016011592 0 0) 0x7f6350004eb0 con 0x7f63640a98f0 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.799+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63640b7520 con 0x7f63640a98f0 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.799+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f63540018c0 con 0x7f63640a98f0 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.799+0000 7f634bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.799+0000 7f634bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.799+0000 7f634bfff640 10 monclient(hunting): dump: 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:03.800 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:03.801 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:03.801 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:03.801 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:03.801 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:03.801 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.799+0000 7f634bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:03.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1211077568 0 0) 0x7f63640b7520 con 0x7f63640a98f0 2024-03-25T18:34:03.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 >> v1:172.21.15.55:6790/0 conn(0x7f63641430c0 legacy=0x7f6364130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 >> v1:172.21.15.133:6789/0 conn(0x7f636413f710 legacy=0x7f63640b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 1 monclient: found mon.a 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63640b6180 con 0x7f63640a98f0 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 10 monclient: _finish_auth 0 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f634bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f636993d640 5 monclient: authenticate success, global_id 4503 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f636993d640 10 monclient: _renew_subs 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f636993d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63640b7a40 con 0x7f63640a98f0 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f636993d640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f636993d640 10 monclient: _renew_subs 2024-03-25T18:34:03.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f636993d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:03.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6364131840 con 0x7f63640a98f0 2024-03-25T18:34:03.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.811+0000 7f636993d640 1 librados: init done 2024-03-25T18:34:03.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.812+0000 7f636993d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:03.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.812+0000 7f636993d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:03.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.812+0000 7f636993d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:03.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:03.812+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6364136f60 con 0x7f63640a98f0 2024-03-25T18:34:04.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.137+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63540035f0 con 0x7f63640a98f0 2024-03-25T18:34:04.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.137+0000 7f634bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:04.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.137+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6354003860 con 0x7f63640a98f0 2024-03-25T18:34:04.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.137+0000 7f634bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:04.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.137+0000 7f634bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:04.138 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.137+0000 7f634bfff640 10 monclient: dump: 2024-03-25T18:34:04.138 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:04.138 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:04.139 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:04.139 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:04.139 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:04.139 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:04.139 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:04.139 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:04.139 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:04.139 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:04.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.137+0000 7f63617fa640 4 set_mon_vals no callback set 2024-03-25T18:34:04.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.138+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f6354003a80 con 0x7f63640a98f0 2024-03-25T18:34:04.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.138+0000 7f63627fc640 10 monclient: get_auth_request con 0x7f635006dd10 auth_method 0 2024-03-25T18:34:04.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.139+0000 7f634bfff640 10 client.4503.objecter ms_handle_connect 0x7f635006dd10 2024-03-25T18:34:04.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.145+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f6354053470 con 0x7f63640a98f0 2024-03-25T18:34:04.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.145+0000 7f634bfff640 10 client.4503.objecter ms_dispatch 0x7f6364000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:04.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.145+0000 7f634bfff640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:04.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.145+0000 7f634bfff640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:04.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.145+0000 7f634bfff640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-25T18:34:04.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.145+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f635404ef80 con 0x7f63640a98f0 2024-03-25T18:34:04.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.145+0000 7f634bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:04.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.145+0000 7f634bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:04.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.363+0000 7f636993d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:04.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.363+0000 7f636993d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:04.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.363+0000 7f636993d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:04.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.363+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63640627b0 con 0x7f63640a98f0 2024-03-25T18:34:04.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.404+0000 7f634bfff640 1 -- 172.21.15.55:0/1643913316 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f6354050eb0 con 0x7f63640a98f0 2024-03-25T18:34:04.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.404+0000 7f634bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:04.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.404+0000 7f634bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:04.405 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:34:04.406 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:04.409 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 10 librados: watch_flush enter 2024-03-25T18:34:04.409 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 10 librados: watch_flush exit 2024-03-25T18:34:04.409 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 10 client.4503.objecter successfully canceled tick 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 >> v1:172.21.15.55:6800/286536064 conn(0x7f635006dd10 legacy=0x7f63500701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 10 monclient: shutdown 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 >> v1:172.21.15.55:6789/0 conn(0x7f63640a98f0 legacy=0x7f6364135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 shutdown_connections 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.407+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 >> 172.21.15.55:0/1643913316 conn(0x7f63640b7ea0 msgr2=0x7f6364074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.408+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 shutdown_connections 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.408+0000 7f636993d640 1 -- 172.21.15.55:0/1643913316 wait complete. 2024-03-25T18:34:04.410 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:04.408+0000 7f636993d640 1 librados: shutdown 2024-03-25T18:34:04.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:11.227 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.342+0000 7f39ac939640 10 monclient: get_monmap_and_config 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.342+0000 7f39ac939640 10 monclient: build_initial_monmap 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.342+0000 7f39ac939640 10 monclient: monmap: 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:34:11.343051+0000 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:34:11.343051+0000 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:34:11.343 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:34:11.344 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.342+0000 7f39ac939640 1 Processor -- start 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 1 -- start start 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 10 monclient: init 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 10 monclient(hunting): picked mon.noname-c con 0x7f39a80a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 10 monclient(hunting): picked mon.noname-b con 0x7f39a813f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 10 monclient(hunting): picked mon.noname-a con 0x7f39a81430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a8142520 con 0x7f39a81430c0 2024-03-25T18:34:11.345 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a80b28a0 con 0x7f39a80a98f0 2024-03-25T18:34:11.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a80b2a30 con 0x7f39a813f710 2024-03-25T18:34:11.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.343+0000 7f39ac939640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:11.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.344+0000 7f39ac939640 10 monclient(hunting): authenticate will time out at 7655.601360s 2024-03-25T18:34:11.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.345+0000 7f39a659c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f39a80a98f0 0x7f39a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:40644/0 (socket says 172.21.15.55:40644) 2024-03-25T18:34:11.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.345+0000 7f39a659c640 1 -- 172.21.15.55:0/3385324889 learned_addr learned my addr 172.21.15.55:0/3385324889 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:11.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.353+0000 7f39a559a640 1 -- 172.21.15.55:0/3385324889 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155329476 0 0) 0x7f39a80b28a0 con 0x7f39a80a98f0 2024-03-25T18:34:11.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.354+0000 7f39a559a640 10 monclient(hunting): _negotiate 2024-03-25T18:34:11.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.354+0000 7f39a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:11.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.354+0000 7f39a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:11.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.354+0000 7f39a559a640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:34:11.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.354+0000 7f39a559a640 1 -- 172.21.15.55:0/3385324889 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3990003520 con 0x7f39a80a98f0 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 1 -- 172.21.15.55:0/3385324889 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2250613985 0 0) 0x7f3990003520 con 0x7f39a80a98f0 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 1 -- 172.21.15.55:0/3385324889 >> v1:172.21.15.55:6790/0 conn(0x7f39a813f710 legacy=0x7f39a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 1 -- 172.21.15.55:0/3385324889 >> v1:172.21.15.55:6789/0 conn(0x7f39a81430c0 legacy=0x7f39a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 10 monclient: _finish_hunting 0 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 1 monclient: found mon.noname-c 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 1 -- 172.21.15.55:0/3385324889 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a80b44c0 con 0x7f39a80a98f0 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 10 monclient: _finish_auth 0 2024-03-25T18:34:11.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 10 monclient: _check_auth_tickets 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 1 -- 172.21.15.55:0/3385324889 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3998001690 con 0x7f39a80a98f0 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.360+0000 7f39a559a640 10 monclient: dump: 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:11.363 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:11.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.361+0000 7f39ac939640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:34:11.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.361+0000 7f39ac939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:11.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.370+0000 7f39a559a640 1 -- 172.21.15.55:0/3385324889 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3998001860 con 0x7f39a80a98f0 2024-03-25T18:34:11.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.370+0000 7f39a559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:11.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.370+0000 7f39a559a640 1 -- 172.21.15.55:0/3385324889 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3998001c70 con 0x7f39a80a98f0 2024-03-25T18:34:11.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.370+0000 7f39ac939640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:11.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.370+0000 7f39ac939640 4 set_mon_vals no callback set 2024-03-25T18:34:11.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.370+0000 7f39a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.370+0000 7f39a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.370+0000 7f39a559a640 10 monclient: dump: 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:11.372 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.371+0000 7f39ac939640 10 monclient: shutdown 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.371+0000 7f39ac939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.371+0000 7f39ac939640 1 -- 172.21.15.55:0/3385324889 >> v1:172.21.15.133:6789/0 conn(0x7f39a80a98f0 legacy=0x7f39a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.371+0000 7f39ac939640 1 -- 172.21.15.55:0/3385324889 shutdown_connections 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.371+0000 7f39ac939640 1 -- 172.21.15.55:0/3385324889 >> 172.21.15.55:0/3385324889 conn(0x7f39a80b7ea0 msgr2=0x7f39a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.371+0000 7f39ac939640 1 -- 172.21.15.55:0/3385324889 shutdown_connections 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.371+0000 7f39ac939640 1 -- 172.21.15.55:0/3385324889 wait complete. 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.372+0000 7f39ac939640 10 monclient: build_initial_monmap 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.372+0000 7f39ac939640 10 monclient: monmap: 2024-03-25T18:34:11.374 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.373+0000 7f39ac939640 1 librados: starting msgr at 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.373+0000 7f39ac939640 1 librados: starting objecter 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.373+0000 7f39ac939640 1 Processor -- start 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.373+0000 7f39ac939640 1 -- start start 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.373+0000 7f39ac939640 1 librados: setting wanted keys 2024-03-25T18:34:11.375 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.373+0000 7f39ac939640 1 librados: calling monclient init 2024-03-25T18:34:11.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.373+0000 7f39ac939640 10 monclient: init 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 10 monclient(hunting): picked mon.noname-c con 0x7f39a80a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 10 monclient(hunting): picked mon.noname-b con 0x7f39a813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 10 monclient(hunting): picked mon.noname-a con 0x7f39a81430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a8071f80 con 0x7f39a81430c0 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a806fb00 con 0x7f39a813f710 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a806fde0 con 0x7f39a80a98f0 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:11.377 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.374+0000 7f39ac939640 10 monclient(hunting): authenticate will time out at 7655.631788s 2024-03-25T18:34:11.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.392+0000 7f39a6d9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f39a81430c0 0x7f39a81e64e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:59378/0 (socket says 172.21.15.55:59378) 2024-03-25T18:34:11.393 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.392+0000 7f39a6d9d640 1 -- 172.21.15.55:0/3702518365 learned_addr learned my addr 172.21.15.55:0/3702518365 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:11.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.394+0000 7f3996ffd640 10 client.?.objecter ms_handle_connect 0x7f39a81430c0 2024-03-25T18:34:11.395 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.394+0000 7f3996ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:11.396 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.396+0000 7f3996ffd640 10 client.?.objecter ms_handle_connect 0x7f39a813f710 2024-03-25T18:34:11.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.396+0000 7f3996ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:11.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.406+0000 7f3996ffd640 10 client.?.objecter ms_handle_connect 0x7f39a80a98f0 2024-03-25T18:34:11.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.406+0000 7f3996ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:11.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.415+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004676605 0 0) 0x7f39a8071f80 con 0x7f39a81430c0 2024-03-25T18:34:11.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.415+0000 7f3996ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:11.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.415+0000 7f3996ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:11.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.415+0000 7f3996ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:11.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.415+0000 7f3996ffd640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:34:11.416 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.415+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3988004eb0 con 0x7f39a81430c0 2024-03-25T18:34:11.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.419+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307508083 0 0) 0x7f39a806fb00 con 0x7f39a813f710 2024-03-25T18:34:11.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.419+0000 7f3996ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:11.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.419+0000 7f3996ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:11.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.419+0000 7f3996ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:11.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.419+0000 7f3996ffd640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:34:11.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.419+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a8071f80 con 0x7f39a813f710 2024-03-25T18:34:11.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.423+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223776803 0 0) 0x7f39a806fde0 con 0x7f39a80a98f0 2024-03-25T18:34:11.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.423+0000 7f3996ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:11.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.423+0000 7f3996ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:11.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.423+0000 7f3996ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:11.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.423+0000 7f3996ffd640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:34:11.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.423+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a806fb00 con 0x7f39a80a98f0 2024-03-25T18:34:11.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.425+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 256032953 0 0) 0x7f3988004eb0 con 0x7f39a81430c0 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.425+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39a806fde0 con 0x7f39a81430c0 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.425+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f39a0001c90 con 0x7f39a81430c0 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.425+0000 7f3996ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.425+0000 7f3996ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.425+0000 7f3996ffd640 10 monclient(hunting): dump: 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:11.426 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:11.427 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:11.427 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:11.427 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:11.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.425+0000 7f3996ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:11.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.428+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2193426553 0 0) 0x7f39a8071f80 con 0x7f39a813f710 2024-03-25T18:34:11.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.428+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3988004eb0 con 0x7f39a813f710 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.428+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f399c001810 con 0x7f39a813f710 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.428+0000 7f3996ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.428+0000 7f3996ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.428+0000 7f3996ffd640 10 monclient(hunting): dump: 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:11.430 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:11.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4244074370 0 0) 0x7f39a806fde0 con 0x7f39a81430c0 2024-03-25T18:34:11.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 >> v1:172.21.15.55:6790/0 conn(0x7f39a80a98f0 legacy=0x7f39a80707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:11.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 >> v1:172.21.15.133:6789/0 conn(0x7f39a813f710 legacy=0x7f39a8071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:11.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:11.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 1 monclient: found mon.a 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a80700c0 con 0x7f39a81430c0 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f3996ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f39ac939640 5 monclient: authenticate success, global_id 4509 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f39ac939640 10 monclient: _renew_subs 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f39ac939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a80703a0 con 0x7f39a81430c0 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f39ac939640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f39ac939640 10 monclient: _renew_subs 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f39ac939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:11.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a8071950 con 0x7f39a81430c0 2024-03-25T18:34:11.435 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.432+0000 7f39ac939640 1 librados: init done 2024-03-25T18:34:11.435 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.433+0000 7f39ac939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:11.435 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.433+0000 7f39ac939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:11.435 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.433+0000 7f39ac939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:11.435 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.433+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a80627b0 con 0x7f39a81430c0 2024-03-25T18:34:11.458 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.457+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39a0003990 con 0x7f39a81430c0 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.458+0000 7f3996ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.458+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f39a0003d60 con 0x7f39a81430c0 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.458+0000 7f3996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.458+0000 7f3996ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.458+0000 7f3996ffd640 10 monclient: dump: 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:11.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.458+0000 7f39a4d99640 4 set_mon_vals no callback set 2024-03-25T18:34:11.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.458+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f39a0003f80 con 0x7f39a81430c0 2024-03-25T18:34:11.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.459+0000 7f39a659c640 10 monclient: get_auth_request con 0x7f398806df70 auth_method 0 2024-03-25T18:34:11.460 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.459+0000 7f3996ffd640 10 client.4509.objecter ms_handle_connect 0x7f398806df70 2024-03-25T18:34:11.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.463+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f39a0054b00 con 0x7f39a81430c0 2024-03-25T18:34:11.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.463+0000 7f3996ffd640 10 client.4509.objecter ms_dispatch 0x7f39a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:11.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.463+0000 7f3996ffd640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:11.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.463+0000 7f3996ffd640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:11.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.463+0000 7f3996ffd640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T18:34:11.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.760+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39a0050610 con 0x7f39a81430c0 2024-03-25T18:34:11.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.760+0000 7f3996ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:11.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:11.760+0000 7f3996ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:12.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.023+0000 7f39ac939640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:12.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.023+0000 7f39ac939640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:12.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.023+0000 7f39ac939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:12.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.023+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39a8074cc0 con 0x7f39a81430c0 2024-03-25T18:34:12.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.113+0000 7f3996ffd640 1 -- 172.21.15.55:0/3702518365 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f39a0052540 con 0x7f39a81430c0 2024-03-25T18:34:12.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.114+0000 7f3996ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:12.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.114+0000 7f3996ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:12.115 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:34:12.115 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:12.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.116+0000 7f39ac939640 10 librados: watch_flush enter 2024-03-25T18:34:12.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.116+0000 7f39ac939640 10 librados: watch_flush exit 2024-03-25T18:34:12.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T18:34:12.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 10 client.4509.objecter successfully canceled tick 2024-03-25T18:34:12.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 >> v1:172.21.15.55:6800/286536064 conn(0x7f398806df70 legacy=0x7f3988070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:12.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 10 monclient: shutdown 2024-03-25T18:34:12.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:12.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 >> v1:172.21.15.55:6789/0 conn(0x7f39a81430c0 legacy=0x7f39a81e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:12.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 shutdown_connections 2024-03-25T18:34:12.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 >> 172.21.15.55:0/3702518365 conn(0x7f39a80b7ea0 msgr2=0x7f39a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:12.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 shutdown_connections 2024-03-25T18:34:12.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 1 -- 172.21.15.55:0/3702518365 wait complete. 2024-03-25T18:34:12.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:12.117+0000 7f39ac939640 1 librados: shutdown 2024-03-25T18:34:12.129 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:15.906 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:06:26.076 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.918 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:06:26.088 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.940 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:06:26.110 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.972 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:26.109 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.975 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:06:26.145 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.984 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:06:26.154 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.987 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:26.124 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:16.021 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:26.158 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:16.021 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:06:26.191 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:18.934 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:19.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.050+0000 7fba94825640 10 monclient: get_monmap_and_config 2024-03-25T18:34:19.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.050+0000 7fba94825640 10 monclient: build_initial_monmap 2024-03-25T18:34:19.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.050+0000 7fba94825640 10 monclient: monmap: 2024-03-25T18:34:19.051 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:19.052 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:19.052 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:34:19.051640+0000 2024-03-25T18:34:19.052 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:34:19.051640+0000 2024-03-25T18:34:19.052 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:19.052 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:19.052 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:19.052 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:34:19.052 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:34:19.052 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:19.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.051+0000 7fba94825640 1 Processor -- start 2024-03-25T18:34:19.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.051+0000 7fba94825640 1 -- start start 2024-03-25T18:34:19.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.051+0000 7fba94825640 10 monclient: init 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 10 monclient(hunting): picked mon.noname-a con 0x7fba900a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 10 monclient(hunting): picked mon.noname-b con 0x7fba9013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 10 monclient(hunting): picked mon.noname-c con 0x7fba901430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba90142520 con 0x7fba900a98f0 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba900b28a0 con 0x7fba901430c0 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba900b2a30 con 0x7fba9013f710 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:19.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.052+0000 7fba94825640 10 monclient(hunting): authenticate will time out at 7663.310222s 2024-03-25T18:34:19.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.058+0000 7fba87fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fba9013f710 0x7fba901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:48618/0 (socket says 172.21.15.55:48618) 2024-03-25T18:34:19.058 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.058+0000 7fba87fff640 1 -- 172.21.15.55:0/3519621579 learned_addr learned my addr 172.21.15.55:0/3519621579 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:19.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.065+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641192046 0 0) 0x7fba900b2a30 con 0x7fba9013f710 2024-03-25T18:34:19.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.065+0000 7fba8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:19.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.065+0000 7fba8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:19.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.065+0000 7fba8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:19.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.065+0000 7fba8dd9b640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:34:19.067 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.065+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c003520 con 0x7fba9013f710 2024-03-25T18:34:19.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.070+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990975830 0 0) 0x7fba900b28a0 con 0x7fba901430c0 2024-03-25T18:34:19.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.070+0000 7fba8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:19.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.070+0000 7fba8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:19.071 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.070+0000 7fba8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:19.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.070+0000 7fba8dd9b640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:34:19.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.070+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba900b2a30 con 0x7fba901430c0 2024-03-25T18:34:19.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2064459918 0 0) 0x7fba7c003520 con 0x7fba9013f710 2024-03-25T18:34:19.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 >> v1:172.21.15.133:6789/0 conn(0x7fba901430c0 legacy=0x7fba9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:19.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 >> v1:172.21.15.55:6789/0 conn(0x7fba900a98f0 legacy=0x7fba900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:19.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:34:19.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 1 monclient: found mon.noname-b 2024-03-25T18:34:19.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:19.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:34:19.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba900b44c0 con 0x7fba9013f710 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fba78001690 con 0x7fba9013f710 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba8dd9b640 10 monclient: dump: 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:19.076 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:19.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba94825640 5 monclient: authenticate success, global_id 4493 2024-03-25T18:34:19.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.074+0000 7fba94825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:19.091 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.090+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba780017f0 con 0x7fba9013f710 2024-03-25T18:34:19.091 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.090+0000 7fba8dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:19.091 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.090+0000 7fba8dd9b640 1 -- 172.21.15.55:0/3519621579 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fba78001c00 con 0x7fba9013f710 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.090+0000 7fba94825640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.090+0000 7fba94825640 4 set_mon_vals no callback set 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.090+0000 7fba8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.090+0000 7fba8dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.090+0000 7fba8dd9b640 10 monclient: dump: 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:19.092 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:19.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.091+0000 7fba94825640 10 monclient: shutdown 2024-03-25T18:34:19.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.091+0000 7fba94825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:19.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.091+0000 7fba94825640 1 -- 172.21.15.55:0/3519621579 >> v1:172.21.15.55:6790/0 conn(0x7fba9013f710 legacy=0x7fba901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:19.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.091+0000 7fba94825640 1 -- 172.21.15.55:0/3519621579 shutdown_connections 2024-03-25T18:34:19.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.091+0000 7fba94825640 1 -- 172.21.15.55:0/3519621579 >> 172.21.15.55:0/3519621579 conn(0x7fba900b7ea0 msgr2=0x7fba900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:19.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.092+0000 7fba94825640 1 -- 172.21.15.55:0/3519621579 shutdown_connections 2024-03-25T18:34:19.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.092+0000 7fba94825640 1 -- 172.21.15.55:0/3519621579 wait complete. 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.093+0000 7fba94825640 10 monclient: build_initial_monmap 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.093+0000 7fba94825640 10 monclient: monmap: 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.093+0000 7fba94825640 1 librados: starting msgr at 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.093+0000 7fba94825640 1 librados: starting objecter 2024-03-25T18:34:19.095 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.093+0000 7fba94825640 1 Processor -- start 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.093+0000 7fba94825640 1 -- start start 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.093+0000 7fba94825640 1 librados: setting wanted keys 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.093+0000 7fba94825640 1 librados: calling monclient init 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.093+0000 7fba94825640 10 monclient: init 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 10 monclient(hunting): picked mon.noname-b con 0x7fba900a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 10 monclient(hunting): picked mon.noname-a con 0x7fba9013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 10 monclient(hunting): picked mon.noname-c con 0x7fba901430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba90137de0 con 0x7fba9013f710 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba90138b20 con 0x7fba900a98f0 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba90139870 con 0x7fba901430c0 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:19.096 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.094+0000 7fba94825640 10 monclient(hunting): authenticate will time out at 7663.351667s 2024-03-25T18:34:19.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.116+0000 7fba8e59c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fba900a98f0 0x7fba901376d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:46572/0 (socket says 172.21.15.55:46572) 2024-03-25T18:34:19.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.116+0000 7fba8e59c640 1 -- 172.21.15.55:0/2014491340 learned_addr learned my addr 172.21.15.55:0/2014491340 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:19.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.117+0000 7fba86ffd640 10 client.?.objecter ms_handle_connect 0x7fba900a98f0 2024-03-25T18:34:19.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.117+0000 7fba86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:19.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.121+0000 7fba86ffd640 10 client.?.objecter ms_handle_connect 0x7fba9013f710 2024-03-25T18:34:19.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.121+0000 7fba86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:19.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.121+0000 7fba86ffd640 10 client.?.objecter ms_handle_connect 0x7fba901430c0 2024-03-25T18:34:19.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.121+0000 7fba86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:19.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.126+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122338263 0 0) 0x7fba90138b20 con 0x7fba900a98f0 2024-03-25T18:34:19.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.126+0000 7fba86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:19.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.126+0000 7fba86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:19.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.126+0000 7fba86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:19.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.126+0000 7fba86ffd640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:34:19.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.126+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba70004eb0 con 0x7fba900a98f0 2024-03-25T18:34:19.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.131+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968706248 0 0) 0x7fba90139870 con 0x7fba901430c0 2024-03-25T18:34:19.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.131+0000 7fba86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:19.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.131+0000 7fba86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:19.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.131+0000 7fba86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:19.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.131+0000 7fba86ffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:34:19.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.131+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba90138b20 con 0x7fba901430c0 2024-03-25T18:34:19.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.136+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442935818 0 0) 0x7fba90137de0 con 0x7fba9013f710 2024-03-25T18:34:19.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.136+0000 7fba86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:19.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.136+0000 7fba86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:19.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.136+0000 7fba86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:19.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.136+0000 7fba86ffd640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:34:19.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.136+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba90139870 con 0x7fba9013f710 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.149+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103836225 0 0) 0x7fba90139870 con 0x7fba9013f710 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.149+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba90137de0 con 0x7fba9013f710 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.149+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fba78001b00 con 0x7fba9013f710 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.149+0000 7fba86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.149+0000 7fba86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.149+0000 7fba86ffd640 10 monclient(hunting): dump: 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:19.150 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:19.151 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:19.151 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:19.151 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:19.151 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:19.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.149+0000 7fba86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.155+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 440215036 0 0) 0x7fba70004eb0 con 0x7fba900a98f0 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.155+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba90139870 con 0x7fba900a98f0 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.155+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fba80001960 con 0x7fba900a98f0 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.155+0000 7fba86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.156+0000 7fba86ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.156+0000 7fba86ffd640 10 monclient(hunting): dump: 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:19.157 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:19.158 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:19.158 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:19.158 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1860685410 0 0) 0x7fba90137de0 con 0x7fba9013f710 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 >> v1:172.21.15.55:6790/0 conn(0x7fba901430c0 legacy=0x7fba901e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 >> v1:172.21.15.133:6789/0 conn(0x7fba900a98f0 legacy=0x7fba901376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 1 monclient: found mon.a 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba900b10b0 con 0x7fba9013f710 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba94825640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba94825640 10 monclient: _renew_subs 2024-03-25T18:34:19.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba94825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:19.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba900b1360 con 0x7fba9013f710 2024-03-25T18:34:19.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba94825640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:19.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba94825640 10 monclient: _renew_subs 2024-03-25T18:34:19.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba94825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:19.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba901d7b70 con 0x7fba9013f710 2024-03-25T18:34:19.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.165+0000 7fba94825640 1 librados: init done 2024-03-25T18:34:19.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.167+0000 7fba94825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:19.168 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.167+0000 7fba94825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:19.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.167+0000 7fba94825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:19.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.167+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba901d7e70 con 0x7fba9013f710 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.476+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba78001c60 con 0x7fba9013f710 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.476+0000 7fba86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.476+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fba78004c30 con 0x7fba9013f710 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.476+0000 7fba86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.476+0000 7fba86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.476+0000 7fba86ffd640 10 monclient: dump: 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:19.477 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:19.478 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:19.478 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:19.478 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:19.478 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:19.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.476+0000 7fba8d59a640 4 set_mon_vals no callback set 2024-03-25T18:34:19.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.477+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fba7801ac00 con 0x7fba9013f710 2024-03-25T18:34:19.478 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.477+0000 7fba8e59c640 10 monclient: get_auth_request con 0x7fba7006df70 auth_method 0 2024-03-25T18:34:19.479 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.478+0000 7fba86ffd640 10 client.4512.objecter ms_handle_connect 0x7fba7006df70 2024-03-25T18:34:19.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.484+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fba780547e0 con 0x7fba9013f710 2024-03-25T18:34:19.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.484+0000 7fba86ffd640 10 client.4512.objecter ms_dispatch 0x7fba90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:19.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.484+0000 7fba86ffd640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:19.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.484+0000 7fba86ffd640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:19.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.484+0000 7fba86ffd640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-25T18:34:19.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.484+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba780502f0 con 0x7fba9013f710 2024-03-25T18:34:19.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.484+0000 7fba86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:19.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.484+0000 7fba86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:19.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.753+0000 7fba94825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:19.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.753+0000 7fba94825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:19.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.753+0000 7fba94825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:19.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.753+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba900b3ef0 con 0x7fba9013f710 2024-03-25T18:34:19.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.825+0000 7fba86ffd640 1 -- 172.21.15.55:0/2014491340 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fba78052220 con 0x7fba9013f710 2024-03-25T18:34:19.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.825+0000 7fba86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:19.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.825+0000 7fba86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:19.826 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:34:19.826 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:19.829 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.827+0000 7fba94825640 10 librados: watch_flush enter 2024-03-25T18:34:19.829 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.827+0000 7fba94825640 10 librados: watch_flush exit 2024-03-25T18:34:19.829 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.827+0000 7fba94825640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-25T18:34:19.829 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.827+0000 7fba94825640 10 client.4512.objecter successfully canceled tick 2024-03-25T18:34:19.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.827+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 >> v1:172.21.15.55:6800/286536064 conn(0x7fba7006df70 legacy=0x7fba70070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:19.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.827+0000 7fba94825640 10 monclient: shutdown 2024-03-25T18:34:19.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.827+0000 7fba94825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:19.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.827+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 >> v1:172.21.15.55:6789/0 conn(0x7fba9013f710 legacy=0x7fba901d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:19.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.828+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 shutdown_connections 2024-03-25T18:34:19.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.828+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 >> 172.21.15.55:0/2014491340 conn(0x7fba900b7ea0 msgr2=0x7fba90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:19.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.828+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 shutdown_connections 2024-03-25T18:34:19.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.828+0000 7fba94825640 1 -- 172.21.15.55:0/2014491340 wait complete. 2024-03-25T18:34:19.830 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:19.828+0000 7fba94825640 1 librados: shutdown 2024-03-25T18:34:19.841 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:26.648 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:26.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.757+0000 7efc854f4640 10 monclient: get_monmap_and_config 2024-03-25T18:34:26.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.757+0000 7efc854f4640 10 monclient: build_initial_monmap 2024-03-25T18:34:26.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.757+0000 7efc854f4640 10 monclient: monmap: 2024-03-25T18:34:26.759 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:26.759 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:26.759 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:34:26.759513+0000 2024-03-25T18:34:26.759 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:34:26.759513+0000 2024-03-25T18:34:26.760 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:26.760 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:26.760 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:26.760 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:34:26.760 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:34:26.760 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.758+0000 7efc854f4640 1 Processor -- start 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.758+0000 7efc854f4640 1 -- start start 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.758+0000 7efc854f4640 10 monclient: init 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 10 monclient(hunting): picked mon.noname-c con 0x7efc800a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 10 monclient(hunting): picked mon.noname-b con 0x7efc8013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 10 monclient(hunting): picked mon.noname-a con 0x7efc801430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc80142520 con 0x7efc801430c0 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc800b28a0 con 0x7efc800a98f0 2024-03-25T18:34:26.761 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc800b2a30 con 0x7efc8013f710 2024-03-25T18:34:26.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:26.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.759+0000 7efc854f4640 10 monclient(hunting): authenticate will time out at 7671.017565s 2024-03-25T18:34:26.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.760+0000 7efc7effd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7efc800a98f0 0x7efc800a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:51570/0 (socket says 172.21.15.55:51570) 2024-03-25T18:34:26.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.760+0000 7efc7effd640 1 -- 172.21.15.55:0/1914809467 learned_addr learned my addr 172.21.15.55:0/1914809467 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:26.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.773+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128449130 0 0) 0x7efc800b28a0 con 0x7efc800a98f0 2024-03-25T18:34:26.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.773+0000 7efc7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.773+0000 7efc7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.773+0000 7efc7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.773+0000 7efc7dffb640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:34:26.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.773+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc68003520 con 0x7efc800a98f0 2024-03-25T18:34:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.774+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214416088 0 0) 0x7efc800b2a30 con 0x7efc8013f710 2024-03-25T18:34:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.774+0000 7efc7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.774+0000 7efc7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.774+0000 7efc7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.774+0000 7efc7dffb640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:34:26.776 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.774+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc800b28a0 con 0x7efc8013f710 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1109266709 0 0) 0x7efc800b28a0 con 0x7efc8013f710 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 >> v1:172.21.15.133:6789/0 conn(0x7efc800a98f0 legacy=0x7efc800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 >> v1:172.21.15.55:6789/0 conn(0x7efc801430c0 legacy=0x7efc8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 1 monclient: found mon.noname-b 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc800b44c0 con 0x7efc8013f710 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 10 monclient: _finish_auth 0 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:26.790 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efc74001670 con 0x7efc8013f710 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc7dffb640 10 monclient: dump: 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc854f4640 5 monclient: authenticate success, global_id 4499 2024-03-25T18:34:26.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.787+0000 7efc854f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.790+0000 7efc7dffb640 1 -- 172.21.15.55:0/1914809467 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc740017d0 con 0x7efc8013f710 2024-03-25T18:34:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.790+0000 7efc7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:26.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.790+0000 7efc854f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.790+0000 7efc854f4640 4 set_mon_vals no callback set 2024-03-25T18:34:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.791+0000 7efc854f4640 10 monclient: shutdown 2024-03-25T18:34:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.791+0000 7efc854f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:26.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.791+0000 7efc854f4640 1 -- 172.21.15.55:0/1914809467 >> v1:172.21.15.55:6790/0 conn(0x7efc8013f710 legacy=0x7efc801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.791+0000 7efc854f4640 1 -- 172.21.15.55:0/1914809467 shutdown_connections 2024-03-25T18:34:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.791+0000 7efc854f4640 1 -- 172.21.15.55:0/1914809467 >> 172.21.15.55:0/1914809467 conn(0x7efc800b7ea0 msgr2=0x7efc800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.791+0000 7efc854f4640 1 -- 172.21.15.55:0/1914809467 shutdown_connections 2024-03-25T18:34:26.794 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.791+0000 7efc854f4640 1 -- 172.21.15.55:0/1914809467 wait complete. 2024-03-25T18:34:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.792+0000 7efc854f4640 10 monclient: build_initial_monmap 2024-03-25T18:34:26.795 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.792+0000 7efc854f4640 10 monclient: monmap: 2024-03-25T18:34:26.795 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:26.795 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:26.795 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:34:26.795 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.792+0000 7efc854f4640 1 librados: starting msgr at 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.792+0000 7efc854f4640 1 librados: starting objecter 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.792+0000 7efc854f4640 1 Processor -- start 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 1 -- start start 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 1 librados: setting wanted keys 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 1 librados: calling monclient init 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 10 monclient: init 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 10 monclient(hunting): picked mon.noname-c con 0x7efc800a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:26.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 10 monclient(hunting): picked mon.noname-a con 0x7efc8013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 10 monclient(hunting): picked mon.noname-b con 0x7efc801430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc800b0d30 con 0x7efc8013f710 2024-03-25T18:34:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc801d82c0 con 0x7efc801430c0 2024-03-25T18:34:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8014e9f0 con 0x7efc800a98f0 2024-03-25T18:34:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:26.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.793+0000 7efc854f4640 10 monclient(hunting): authenticate will time out at 7671.051954s 2024-03-25T18:34:26.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.797+0000 7efc7f7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7efc801430c0 0x7efc801d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:51582/0 (socket says 172.21.15.55:51582) 2024-03-25T18:34:26.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.797+0000 7efc7f7fe640 1 -- 172.21.15.55:0/2073175541 learned_addr learned my addr 172.21.15.55:0/2073175541 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.798+0000 7efc677fe640 10 client.?.objecter ms_handle_connect 0x7efc801430c0 2024-03-25T18:34:26.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.798+0000 7efc677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:26.803 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.801+0000 7efc677fe640 10 client.?.objecter ms_handle_connect 0x7efc800a98f0 2024-03-25T18:34:26.803 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.801+0000 7efc677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:26.808 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.806+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418787334 0 0) 0x7efc801d82c0 con 0x7efc801430c0 2024-03-25T18:34:26.808 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.806+0000 7efc677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.807+0000 7efc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.807+0000 7efc677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.807+0000 7efc677fe640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:34:26.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.807+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc6c004eb0 con 0x7efc801430c0 2024-03-25T18:34:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.809+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663150933 0 0) 0x7efc8014e9f0 con 0x7efc800a98f0 2024-03-25T18:34:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.809+0000 7efc677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.809+0000 7efc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.809+0000 7efc677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.809+0000 7efc677fe640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:34:26.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.809+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc801d82c0 con 0x7efc800a98f0 2024-03-25T18:34:26.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.810+0000 7efc677fe640 10 client.?.objecter ms_handle_connect 0x7efc8013f710 2024-03-25T18:34:26.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.810+0000 7efc677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:26.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.814+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4267977316 0 0) 0x7efc6c004eb0 con 0x7efc801430c0 2024-03-25T18:34:26.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.814+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc8014e9f0 con 0x7efc801430c0 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.816+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efc780043a0 con 0x7efc801430c0 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 10 monclient(hunting): dump: 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:26.819 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:26.820 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:26.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:26.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457736415 0 0) 0x7efc800b0d30 con 0x7efc8013f710 2024-03-25T18:34:26.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:26.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:26.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:26.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:34:26.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.817+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc6c004eb0 con 0x7efc8013f710 2024-03-25T18:34:26.824 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.822+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500907364 0 0) 0x7efc801d82c0 con 0x7efc800a98f0 2024-03-25T18:34:26.824 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.822+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc800b0d30 con 0x7efc800a98f0 2024-03-25T18:34:26.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1981222333 0 0) 0x7efc8014e9f0 con 0x7efc801430c0 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 >> v1:172.21.15.55:6790/0 conn(0x7efc800a98f0 legacy=0x7efc801d4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 >> v1:172.21.15.55:6789/0 conn(0x7efc8013f710 legacy=0x7efc801d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 1 monclient: found mon.b 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc800b14f0 con 0x7efc801430c0 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc854f4640 5 monclient: authenticate success, global_id 4492 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc854f4640 10 monclient: _renew_subs 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc854f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:26.826 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc800b1150 con 0x7efc801430c0 2024-03-25T18:34:26.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc854f4640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:26.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc854f4640 10 monclient: _renew_subs 2024-03-25T18:34:26.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc854f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:26.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8014e130 con 0x7efc801430c0 2024-03-25T18:34:26.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.823+0000 7efc854f4640 1 librados: init done 2024-03-25T18:34:26.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.824+0000 7efc854f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:26.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.824+0000 7efc854f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:26.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.824+0000 7efc854f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:26.827 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.824+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc800b1340 con 0x7efc801430c0 2024-03-25T18:34:26.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.843+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc78001250 con 0x7efc801430c0 2024-03-25T18:34:26.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.843+0000 7efc677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:26.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.843+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efc780016a0 con 0x7efc801430c0 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.843+0000 7efc677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.843+0000 7efc677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.843+0000 7efc677fe640 10 monclient: dump: 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:26.846 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.843+0000 7efc7d7fa640 4 set_mon_vals no callback set 2024-03-25T18:34:26.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.844+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7efc780018c0 con 0x7efc801430c0 2024-03-25T18:34:26.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.845+0000 7efc7effd640 10 monclient: get_auth_request con 0x7efc6c06e0d0 auth_method 0 2024-03-25T18:34:26.847 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.845+0000 7efc677fe640 10 client.4492.objecter ms_handle_connect 0x7efc6c06e0d0 2024-03-25T18:34:26.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.852+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7efc78054ad0 con 0x7efc801430c0 2024-03-25T18:34:26.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.852+0000 7efc677fe640 10 client.4492.objecter ms_dispatch 0x7efc80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:26.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.852+0000 7efc677fe640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:26.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.852+0000 7efc677fe640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:26.854 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:26.852+0000 7efc677fe640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-25T18:34:27.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.119+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc780505e0 con 0x7efc801430c0 2024-03-25T18:34:27.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.119+0000 7efc677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:27.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.119+0000 7efc677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:27.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.344+0000 7efc854f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:27.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.344+0000 7efc854f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:27.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.344+0000 7efc854f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:27.346 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.344+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc80069f40 con 0x7efc801430c0 2024-03-25T18:34:27.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc7f7fe640 1 -- 172.21.15.55:0/2073175541 >> v1:172.21.15.133:6789/0 conn(0x7efc801430c0 legacy=0x7efc801d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:34:27.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc7f7fe640 1 -- 172.21.15.55:0/2073175541 >> v1:172.21.15.133:6789/0 conn(0x7efc801430c0 legacy=0x7efc801d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:34:27.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc7f7fe640 1 --1- 172.21.15.55:0/2073175541 >> v1:172.21.15.133:6789/0 conn(0x7efc801430c0 0x7efc801d41e0 :-1 s=OPENED pgs=44109 cs=1 l=1).handle_message read tag failed 2024-03-25T18:34:27.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc7f7fe640 1 --1- 172.21.15.55:0/2073175541 >> v1:172.21.15.133:6789/0 conn(0x7efc801430c0 0x7efc801d41e0 :-1 s=OPENED pgs=44109 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:34:27.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.133:6789/0 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 >> v1:172.21.15.133:6789/0 conn(0x7efc801430c0 legacy=0x7efc801d41e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 10 monclient(hunting): picked mon.c con 0x7efc800a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 10 monclient(hunting): picked mon.b con 0x7efc6c076b30 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 10 monclient(hunting): picked mon.a con 0x7efc8013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efc8014e9f0 con 0x7efc8013f710 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efc801d82c0 con 0x7efc6c076b30 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efc6c080840 con 0x7efc800a98f0 2024-03-25T18:34:27.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.362+0000 7efc677fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:27.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.374+0000 7efc677fe640 10 client.4492.objecter ms_handle_connect 0x7efc800a98f0 2024-03-25T18:34:27.376 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.374+0000 7efc677fe640 10 client.4492.objecter resend_mon_ops 2024-03-25T18:34:27.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.377+0000 7efc677fe640 10 client.4492.objecter ms_handle_connect 0x7efc6c076b30 2024-03-25T18:34:27.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.377+0000 7efc677fe640 10 client.4492.objecter resend_mon_ops 2024-03-25T18:34:27.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.380+0000 7efc677fe640 10 client.4492.objecter ms_handle_connect 0x7efc8013f710 2024-03-25T18:34:27.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.380+0000 7efc677fe640 10 client.4492.objecter resend_mon_ops 2024-03-25T18:34:27.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.380+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033271761 0 0) 0x7efc6c080840 con 0x7efc800a98f0 2024-03-25T18:34:27.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.380+0000 7efc677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:27.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.380+0000 7efc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:27.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.380+0000 7efc677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:34:27.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.380+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.55:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7efc6c07f810 con 0x7efc800a98f0 2024-03-25T18:34:27.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.392+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568630399 0 0) 0x7efc8014e9f0 con 0x7efc8013f710 2024-03-25T18:34:27.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.392+0000 7efc677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:27.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.392+0000 7efc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:27.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.392+0000 7efc677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:34:27.394 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.392+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.55:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7efc6c080840 con 0x7efc8013f710 2024-03-25T18:34:27.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3892751223 0 0) 0x7efc6c080840 con 0x7efc8013f710 2024-03-25T18:34:27.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 >> v1:172.21.15.55:6790/0 conn(0x7efc800a98f0 legacy=0x7efc6c075160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:27.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 >> v1:172.21.15.133:6789/0 conn(0x7efc6c076b30 legacy=0x7efc6c078fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:27.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:27.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 1 monclient: found mon.a 2024-03-25T18:34:27.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:27.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:27.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7efc6c081aa0 con 0x7efc8013f710 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc8014e9f0 con 0x7efc8013f710 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7efc700058d0 con 0x7efc8013f710 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.400+0000 7efc677fe640 10 monclient: dump: 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:27.403 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:27.404 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:27.404 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:27.404 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:27.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.457+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc70001cc0 con 0x7efc8013f710 2024-03-25T18:34:27.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.457+0000 7efc677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:27.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.457+0000 7efc7d7fa640 4 set_mon_vals no callback set 2024-03-25T18:34:27.459 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.458+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7efc70005ab0 con 0x7efc8013f710 2024-03-25T18:34:27.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.462+0000 7efc677fe640 1 -- 172.21.15.55:0/2073175541 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7efc70059ba0 con 0x7efc8013f710 2024-03-25T18:34:27.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.462+0000 7efc677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:27.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.462+0000 7efc677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:27.464 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:34:27.464 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:27.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 10 librados: watch_flush enter 2024-03-25T18:34:27.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 10 librados: watch_flush exit 2024-03-25T18:34:27.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 10 client.4492.objecter successfully canceled tick 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 >> v1:172.21.15.55:6800/286536064 conn(0x7efc6c06e0d0 legacy=0x7efc6c070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 10 monclient: shutdown 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 >> v1:172.21.15.55:6789/0 conn(0x7efc8013f710 legacy=0x7efc6c07cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 shutdown_connections 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 >> 172.21.15.55:0/2073175541 conn(0x7efc800b7ea0 msgr2=0x7efc80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 shutdown_connections 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 1 -- 172.21.15.55:0/2073175541 wait complete. 2024-03-25T18:34:27.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:27.465+0000 7efc854f4640 1 librados: shutdown 2024-03-25T18:34:27.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:34.285 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:34.400 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.399+0000 7f64eecf1640 10 monclient: get_monmap_and_config 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.399+0000 7f64eecf1640 10 monclient: build_initial_monmap 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.399+0000 7f64eecf1640 10 monclient: monmap: 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:34:34.400640+0000 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:34:34.400640+0000 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:34:34.401 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:34:34.402 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:34.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.400+0000 7f64eecf1640 1 Processor -- start 2024-03-25T18:34:34.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.400+0000 7f64eecf1640 1 -- start start 2024-03-25T18:34:34.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.400+0000 7f64eecf1640 10 monclient: init 2024-03-25T18:34:34.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:34.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:34.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 10 monclient(hunting): picked mon.noname-c con 0x7f64e80b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:34.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 10 monclient(hunting): picked mon.noname-a con 0x7f64e80b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:34.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 10 monclient(hunting): picked mon.noname-b con 0x7f64e814be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:34.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80af940 con 0x7f64e80b1730 2024-03-25T18:34:34.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80b0a70 con 0x7f64e80b4d60 2024-03-25T18:34:34.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80b0d50 con 0x7f64e814be90 2024-03-25T18:34:34.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:34.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.401+0000 7f64eecf1640 10 monclient(hunting): authenticate will time out at 7678.659158s 2024-03-25T18:34:34.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.406+0000 7f64e7fff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f64e80b1730 0x7f64e80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41474/0 (socket says 172.21.15.55:41474) 2024-03-25T18:34:34.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.406+0000 7f64e7fff640 1 -- 172.21.15.55:0/4263272064 learned_addr learned my addr 172.21.15.55:0/4263272064 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:34.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.411+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437891880 0 0) 0x7f64e80af940 con 0x7f64e80b1730 2024-03-25T18:34:34.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.411+0000 7f64e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:34.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.411+0000 7f64e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:34.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.411+0000 7f64e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:34.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.411+0000 7f64e77fe640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:34:34.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.412+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d8003520 con 0x7f64e80b1730 2024-03-25T18:34:34.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.419+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808824365 0 0) 0x7f64e80b0d50 con 0x7f64e814be90 2024-03-25T18:34:34.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.419+0000 7f64e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:34.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.419+0000 7f64e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:34.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.419+0000 7f64e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:34.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.419+0000 7f64e77fe640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:34:34.420 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.419+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e80af940 con 0x7f64e814be90 2024-03-25T18:34:34.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3923351014 0 0) 0x7f64d8003520 con 0x7f64e80b1730 2024-03-25T18:34:34.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 >> v1:172.21.15.55:6790/0 conn(0x7f64e814be90 legacy=0x7f64e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:34.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 >> v1:172.21.15.133:6789/0 conn(0x7f64e80b4d60 legacy=0x7f64e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:34.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:34.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 1 monclient: found mon.noname-a 2024-03-25T18:34:34.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:34.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:34:34.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e80b1030 con 0x7f64e80b1730 2024-03-25T18:34:34.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f64d4001690 con 0x7f64e80b1730 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64eecf1640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64eecf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.429+0000 7f64e77fe640 10 monclient: dump: 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:34.432 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:34.433 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:34.433 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:34.433 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:34.433 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:34.433 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:34.433 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.430+0000 7f64eecf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:34.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.445+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64d40017f0 con 0x7f64e80b1730 2024-03-25T18:34:34.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.445+0000 7f64e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:34.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.445+0000 7f64e77fe640 1 -- 172.21.15.55:0/4263272064 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f64d4001c00 con 0x7f64e80b1730 2024-03-25T18:34:34.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.445+0000 7f64e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:34.446 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.445+0000 7f64e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.445+0000 7f64e77fe640 10 monclient: dump: 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:34.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.445+0000 7f64eecf1640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:34.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.445+0000 7f64eecf1640 4 set_mon_vals no callback set 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.446+0000 7f64eecf1640 10 monclient: shutdown 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.446+0000 7f64eecf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.446+0000 7f64eecf1640 1 -- 172.21.15.55:0/4263272064 >> v1:172.21.15.55:6789/0 conn(0x7f64e80b1730 legacy=0x7f64e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.446+0000 7f64eecf1640 1 -- 172.21.15.55:0/4263272064 shutdown_connections 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.446+0000 7f64eecf1640 1 -- 172.21.15.55:0/4263272064 >> 172.21.15.55:0/4263272064 conn(0x7f64e80617a0 msgr2=0x7f64e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.446+0000 7f64eecf1640 1 -- 172.21.15.55:0/4263272064 shutdown_connections 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.446+0000 7f64eecf1640 1 -- 172.21.15.55:0/4263272064 wait complete. 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.447+0000 7f64eecf1640 10 monclient: build_initial_monmap 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.447+0000 7f64eecf1640 10 monclient: monmap: 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:34.450 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.447+0000 7f64eecf1640 1 librados: starting msgr at 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.447+0000 7f64eecf1640 1 librados: starting objecter 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.447+0000 7f64eecf1640 1 Processor -- start 2024-03-25T18:34:34.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.447+0000 7f64eecf1640 1 -- start start 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.447+0000 7f64eecf1640 1 librados: setting wanted keys 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.447+0000 7f64eecf1640 1 librados: calling monclient init 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.447+0000 7f64eecf1640 10 monclient: init 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 10 monclient(hunting): picked mon.noname-b con 0x7f64e80b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 10 monclient(hunting): picked mon.noname-c con 0x7f64e80b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 10 monclient(hunting): picked mon.noname-a con 0x7f64e814be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e8141f10 con 0x7f64e814be90 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e8140da0 con 0x7f64e80b1730 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e8138860 con 0x7f64e80b4d60 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:34.452 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.448+0000 7f64eecf1640 10 monclient(hunting): authenticate will time out at 7678.706101s 2024-03-25T18:34:34.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.450+0000 7f64eca8d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f64e80b1730 0x7f64e81405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:54942/0 (socket says 172.21.15.55:54942) 2024-03-25T18:34:34.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.450+0000 7f64eca8d640 1 -- 172.21.15.55:0/1076711855 learned_addr learned my addr 172.21.15.55:0/1076711855 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:34.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.451+0000 7f64e57fa640 10 client.?.objecter ms_handle_connect 0x7f64e80b1730 2024-03-25T18:34:34.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.451+0000 7f64e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:34.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.455+0000 7f64e57fa640 10 client.?.objecter ms_handle_connect 0x7f64e80b4d60 2024-03-25T18:34:34.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.455+0000 7f64e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:34.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.524+0000 7f64e57fa640 10 client.?.objecter ms_handle_connect 0x7f64e814be90 2024-03-25T18:34:34.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.524+0000 7f64e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:34.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.535+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956780094 0 0) 0x7f64e8141f10 con 0x7f64e814be90 2024-03-25T18:34:34.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.535+0000 7f64e57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:34.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.535+0000 7f64e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:34.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.535+0000 7f64e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:34.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.535+0000 7f64e57fa640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:34:34.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.535+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64cc004eb0 con 0x7f64e814be90 2024-03-25T18:34:34.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.543+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916687547 0 0) 0x7f64e8140da0 con 0x7f64e80b1730 2024-03-25T18:34:34.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.543+0000 7f64e57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:34.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.543+0000 7f64e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:34.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.543+0000 7f64e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:34.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.543+0000 7f64e57fa640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:34:34.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.543+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e8141f10 con 0x7f64e80b1730 2024-03-25T18:34:34.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.551+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3164748308 0 0) 0x7f64cc004eb0 con 0x7f64e814be90 2024-03-25T18:34:34.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.551+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64e8140da0 con 0x7f64e814be90 2024-03-25T18:34:34.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.552+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f64e0001bd0 con 0x7f64e814be90 2024-03-25T18:34:34.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.552+0000 7f64e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:34.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.552+0000 7f64e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:34.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.552+0000 7f64e57fa640 10 monclient(hunting): dump: 2024-03-25T18:34:34.553 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:34.553 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:34.553 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:34.554 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:34.554 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:34.554 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:34.554 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:34.554 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:34.554 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:34.554 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:34.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.552+0000 7f64e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:34.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.553+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418024984 0 0) 0x7f64e8138860 con 0x7f64e80b4d60 2024-03-25T18:34:34.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.553+0000 7f64e57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:34.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.553+0000 7f64e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:34.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.553+0000 7f64e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:34.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.553+0000 7f64e57fa640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:34:34.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.553+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64cc004eb0 con 0x7f64e80b4d60 2024-03-25T18:34:34.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2912776690 0 0) 0x7f64e8140da0 con 0x7f64e814be90 2024-03-25T18:34:34.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 >> v1:172.21.15.55:6790/0 conn(0x7f64e80b4d60 legacy=0x7f64e81d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:34.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 >> v1:172.21.15.133:6789/0 conn(0x7f64e80b1730 legacy=0x7f64e81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:34.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:34.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 1 monclient: found mon.a 2024-03-25T18:34:34.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:34.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:34.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8137f50 con 0x7f64e814be90 2024-03-25T18:34:34.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:34.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:34.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:34.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64eecf1640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:34:34.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64eecf1640 10 monclient: _renew_subs 2024-03-25T18:34:34.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64eecf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:34.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e81d8080 con 0x7f64e814be90 2024-03-25T18:34:34.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64eecf1640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:34.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64eecf1640 10 monclient: _renew_subs 2024-03-25T18:34:34.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64eecf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:34.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e81d85b0 con 0x7f64e814be90 2024-03-25T18:34:34.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.561+0000 7f64eecf1640 1 librados: init done 2024-03-25T18:34:34.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.562+0000 7f64eecf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:34.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.562+0000 7f64eecf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:34.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.562+0000 7f64eecf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:34.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.562+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e81381a0 con 0x7f64e814be90 2024-03-25T18:34:34.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.890+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64e0002080 con 0x7f64e814be90 2024-03-25T18:34:34.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.890+0000 7f64e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:34.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.890+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f64e0003b30 con 0x7f64e814be90 2024-03-25T18:34:34.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.890+0000 7f64e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:34.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.890+0000 7f64e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:34.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.890+0000 7f64e57fa640 10 monclient: dump: 2024-03-25T18:34:34.892 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:34.892 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:34.892 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:34.893 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:34.893 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:34.893 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:34.893 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:34.893 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:34.893 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:34.893 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:34.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.890+0000 7f64e6ffd640 4 set_mon_vals no callback set 2024-03-25T18:34:34.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.891+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f64e0019b00 con 0x7f64e814be90 2024-03-25T18:34:34.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.892+0000 7f64eca8d640 10 monclient: get_auth_request con 0x7f64cc06ddb0 auth_method 0 2024-03-25T18:34:34.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.892+0000 7f64e57fa640 10 client.4521.objecter ms_handle_connect 0x7f64cc06ddb0 2024-03-25T18:34:34.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.897+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f64e00537f0 con 0x7f64e814be90 2024-03-25T18:34:34.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.897+0000 7f64e57fa640 10 client.4521.objecter ms_dispatch 0x7f64e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:34.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.897+0000 7f64e57fa640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:34.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.897+0000 7f64e57fa640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:34.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.897+0000 7f64e57fa640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T18:34:34.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.897+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64e004f130 con 0x7f64e814be90 2024-03-25T18:34:34.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.897+0000 7f64e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:34.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:34.897+0000 7f64e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:35.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.146+0000 7f64eecf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:35.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.146+0000 7f64eecf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:35.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.146+0000 7f64eecf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:34:35.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.146+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64e80627b0 con 0x7f64e814be90 2024-03-25T18:34:35.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.172+0000 7f64e57fa640 1 -- 172.21.15.55:0/1076711855 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f64e0051230 con 0x7f64e814be90 2024-03-25T18:34:35.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.172+0000 7f64e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:35.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.172+0000 7f64e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:35.173 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:34:35.174 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 10 librados: watch_flush enter 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 10 librados: watch_flush exit 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 10 client.4521.objecter successfully canceled tick 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 >> v1:172.21.15.55:6800/286536064 conn(0x7f64cc06ddb0 legacy=0x7f64cc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 10 monclient: shutdown 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 >> v1:172.21.15.55:6789/0 conn(0x7f64e814be90 legacy=0x7f64e81e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 shutdown_connections 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 >> 172.21.15.55:0/1076711855 conn(0x7f64e80617a0 msgr2=0x7f64e80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 shutdown_connections 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 1 -- 172.21.15.55:0/1076711855 wait complete. 2024-03-25T18:34:35.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:35.174+0000 7f64eecf1640 1 librados: shutdown 2024-03-25T18:34:35.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:35.930 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:06:46.100 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.976 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:46.113 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.978 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:46.116 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.983 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:06:46.153 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:36.008 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:06:46.178 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:36.009 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:06:46.146 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:36.052 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:06:46.222 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:36.056 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:06:46.226 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:36.095 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:06:46.265 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:41.995 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.100+0000 7fbd5eb79640 10 monclient: get_monmap_and_config 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.100+0000 7fbd5eb79640 10 monclient: build_initial_monmap 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.100+0000 7fbd5eb79640 10 monclient: monmap: 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:34:42.101167+0000 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:34:42.101167+0000 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:34:42.101 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:34:42.102 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 1 Processor -- start 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 1 -- start start 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 10 monclient: init 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 10 monclient(hunting): picked mon.noname-c con 0x7fbd580b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 10 monclient(hunting): picked mon.noname-b con 0x7fbd580b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 10 monclient(hunting): picked mon.noname-a con 0x7fbd5814be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580af940 con 0x7fbd5814be90 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580b0a70 con 0x7fbd580b4d60 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580b0d50 con 0x7fbd580b1730 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:42.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.101+0000 7fbd5eb79640 10 monclient(hunting): authenticate will time out at 7686.359046s 2024-03-25T18:34:42.123 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.123+0000 7fbd5c915640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fbd580b4d60 0x7fbd5813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:55952/0 (socket says 172.21.15.55:55952) 2024-03-25T18:34:42.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.123+0000 7fbd5c915640 1 -- 172.21.15.55:0/31901285 learned_addr learned my addr 172.21.15.55:0/31901285 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:42.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.135+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747800530 0 0) 0x7fbd580b0a70 con 0x7fbd580b4d60 2024-03-25T18:34:42.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.135+0000 7fbd577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:42.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.135+0000 7fbd577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:42.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.135+0000 7fbd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:42.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.135+0000 7fbd577fe640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:34:42.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.135+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd48003520 con 0x7fbd580b4d60 2024-03-25T18:34:42.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.145+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308704134 0 0) 0x7fbd580b0d50 con 0x7fbd580b1730 2024-03-25T18:34:42.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.145+0000 7fbd577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:42.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.145+0000 7fbd577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:42.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.145+0000 7fbd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:42.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.145+0000 7fbd577fe640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:34:42.146 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.145+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd580b0a70 con 0x7fbd580b1730 2024-03-25T18:34:42.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 524698112 0 0) 0x7fbd48003520 con 0x7fbd580b4d60 2024-03-25T18:34:42.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 >> v1:172.21.15.55:6790/0 conn(0x7fbd580b1730 legacy=0x7fbd580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.147 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 >> v1:172.21.15.55:6789/0 conn(0x7fbd5814be90 legacy=0x7fbd5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 1 monclient: found mon.noname-c 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd580b1030 con 0x7fbd580b4d60 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbd4c001690 con 0x7fbd580b4d60 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd5eb79640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd5eb79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd577fe640 10 monclient: dump: 2024-03-25T18:34:42.148 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:42.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.146+0000 7fbd5eb79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd4c0017f0 con 0x7fbd580b4d60 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd577fe640 1 -- 172.21.15.55:0/31901285 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbd4c001c00 con 0x7fbd580b4d60 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd577fe640 10 monclient: dump: 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:42.171 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:42.172 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:42.172 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:42.172 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:42.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd5eb79640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:42.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd5eb79640 4 set_mon_vals no callback set 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd5eb79640 10 monclient: shutdown 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd5eb79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.170+0000 7fbd5eb79640 1 -- 172.21.15.55:0/31901285 >> v1:172.21.15.133:6789/0 conn(0x7fbd580b4d60 legacy=0x7fbd5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.171+0000 7fbd5eb79640 1 -- 172.21.15.55:0/31901285 shutdown_connections 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.171+0000 7fbd5eb79640 1 -- 172.21.15.55:0/31901285 >> 172.21.15.55:0/31901285 conn(0x7fbd580617a0 msgr2=0x7fbd58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.171+0000 7fbd5eb79640 1 -- 172.21.15.55:0/31901285 shutdown_connections 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.171+0000 7fbd5eb79640 1 -- 172.21.15.55:0/31901285 wait complete. 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.171+0000 7fbd5eb79640 10 monclient: build_initial_monmap 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.171+0000 7fbd5eb79640 10 monclient: monmap: 2024-03-25T18:34:42.173 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.172+0000 7fbd5eb79640 1 librados: starting msgr at 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.172+0000 7fbd5eb79640 1 librados: starting objecter 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.172+0000 7fbd5eb79640 1 Processor -- start 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.172+0000 7fbd5eb79640 1 -- start start 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.172+0000 7fbd5eb79640 1 librados: setting wanted keys 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.172+0000 7fbd5eb79640 1 librados: calling monclient init 2024-03-25T18:34:42.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.172+0000 7fbd5eb79640 10 monclient: init 2024-03-25T18:34:42.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:42.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:42.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 10 monclient(hunting): picked mon.noname-b con 0x7fbd580b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:42.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 10 monclient(hunting): picked mon.noname-a con 0x7fbd580b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:42.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 10 monclient(hunting): picked mon.noname-c con 0x7fbd5814be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:42.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd58141f40 con 0x7fbd580b4d60 2024-03-25T18:34:42.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd58142360 con 0x7fbd580b1730 2024-03-25T18:34:42.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd581424f0 con 0x7fbd5814be90 2024-03-25T18:34:42.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:42.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.173+0000 7fbd5eb79640 10 monclient(hunting): authenticate will time out at 7686.430775s 2024-03-25T18:34:42.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.177+0000 7fbd5c915640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fbd580b1730 0x7fbd581405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:55980/0 (socket says 172.21.15.55:55980) 2024-03-25T18:34:42.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.177+0000 7fbd5c915640 1 -- 172.21.15.55:0/4145529667 learned_addr learned my addr 172.21.15.55:0/4145529667 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:42.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.178+0000 7fbd557fa640 10 client.?.objecter ms_handle_connect 0x7fbd580b1730 2024-03-25T18:34:42.178 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.178+0000 7fbd557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:42.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.182+0000 7fbd557fa640 10 client.?.objecter ms_handle_connect 0x7fbd5814be90 2024-03-25T18:34:42.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.182+0000 7fbd557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:42.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.188+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906445772 0 0) 0x7fbd580af940 con 0x7fbd5814be90 2024-03-25T18:34:42.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.188+0000 7fbd557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:42.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.188+0000 7fbd557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:42.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.188+0000 7fbd557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:42.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.188+0000 7fbd557fa640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:34:42.189 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.188+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd3c004eb0 con 0x7fbd5814be90 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.189+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667091997 0 0) 0x7fbd58142360 con 0x7fbd580b1730 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.189+0000 7fbd557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.189+0000 7fbd557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.189+0000 7fbd557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.189+0000 7fbd557fa640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:34:42.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.189+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd580af940 con 0x7fbd580b1730 2024-03-25T18:34:42.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.194+0000 7fbd557fa640 10 client.?.objecter ms_handle_connect 0x7fbd580b4d60 2024-03-25T18:34:42.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.194+0000 7fbd557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:42.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.198+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2081069021 0 0) 0x7fbd3c004eb0 con 0x7fbd5814be90 2024-03-25T18:34:42.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.198+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd58142360 con 0x7fbd5814be90 2024-03-25T18:34:42.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.198+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbd500019f0 con 0x7fbd5814be90 2024-03-25T18:34:42.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.198+0000 7fbd557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:42.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.198+0000 7fbd557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:42.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.198+0000 7fbd557fa640 10 monclient(hunting): dump: 2024-03-25T18:34:42.199 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:42.199 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:42.199 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:42.200 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:42.200 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:42.200 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:42.200 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:42.200 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:42.200 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:42.200 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:42.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.198+0000 7fbd557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:42.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2762471328 0 0) 0x7fbd58142360 con 0x7fbd5814be90 2024-03-25T18:34:42.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 >> v1:172.21.15.133:6789/0 conn(0x7fbd580b1730 legacy=0x7fbd581405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 >> v1:172.21.15.55:6789/0 conn(0x7fbd580b4d60 legacy=0x7fbd581d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:42.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 1 monclient: found mon.c 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58140d00 con 0x7fbd5814be90 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd5eb79640 5 monclient: authenticate success, global_id 4514 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd5eb79640 10 monclient: _renew_subs 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd5eb79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd58138150 con 0x7fbd5814be90 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd5eb79640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd5eb79640 10 monclient: _renew_subs 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd5eb79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:34:42.206 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd581d8520 con 0x7fbd5814be90 2024-03-25T18:34:42.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.204+0000 7fbd5eb79640 1 librados: init done 2024-03-25T18:34:42.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.205+0000 7fbd5eb79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:42.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.205+0000 7fbd5eb79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:42.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.205+0000 7fbd5eb79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:34:42.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.205+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd58140f50 con 0x7fbd5814be90 2024-03-25T18:34:42.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.216+0000 7fbd5d116640 1 -- 172.21.15.55:0/4145529667 >> v1:172.21.15.55:6790/0 conn(0x7fbd5814be90 legacy=0x7fbd581e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:34:42.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.216+0000 7fbd5d116640 1 -- 172.21.15.55:0/4145529667 >> v1:172.21.15.55:6790/0 conn(0x7fbd5814be90 legacy=0x7fbd581e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:34:42.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.216+0000 7fbd5d116640 1 --1- 172.21.15.55:0/4145529667 >> v1:172.21.15.55:6790/0 conn(0x7fbd5814be90 0x7fbd581e2290 :-1 s=OPENED pgs=45240 cs=1 l=1).handle_message read tag failed 2024-03-25T18:34:42.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.216+0000 7fbd5d116640 1 --1- 172.21.15.55:0/4145529667 >> v1:172.21.15.55:6790/0 conn(0x7fbd5814be90 0x7fbd581e2290 :-1 s=OPENED pgs=45240 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:34:42.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.216+0000 7fbd557fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.55:6790/0 2024-03-25T18:34:42.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.216+0000 7fbd557fa640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:42.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.216+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 >> v1:172.21.15.55:6790/0 conn(0x7fbd5814be90 legacy=0x7fbd581e2290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:34:42.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.216+0000 7fbd557fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:42.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.217+0000 7fbd557fa640 10 monclient(hunting): picked mon.a con 0x7fbd580b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:42.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.217+0000 7fbd557fa640 10 monclient(hunting): picked mon.c con 0x7fbd3c006af0 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:42.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.217+0000 7fbd557fa640 10 monclient(hunting): picked mon.b con 0x7fbd580b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:42.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.217+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbd58142360 con 0x7fbd580b4d60 2024-03-25T18:34:42.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.217+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbd3c004eb0 con 0x7fbd580b1730 2024-03-25T18:34:42.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.217+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbd3c0139c0 con 0x7fbd3c006af0 2024-03-25T18:34:42.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.217+0000 7fbd557fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:42.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.222+0000 7fbd557fa640 10 client.4514.objecter ms_handle_connect 0x7fbd3c006af0 2024-03-25T18:34:42.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.222+0000 7fbd557fa640 10 client.4514.objecter resend_mon_ops 2024-03-25T18:34:42.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.227+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149199803 0 0) 0x7fbd3c0139c0 con 0x7fbd3c006af0 2024-03-25T18:34:42.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.227+0000 7fbd557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:42.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.227+0000 7fbd557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:42.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.227+0000 7fbd557fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:34:42.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.227+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fbd3c013b50 con 0x7fbd3c006af0 2024-03-25T18:34:42.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.229+0000 7fbd557fa640 10 client.4514.objecter ms_handle_connect 0x7fbd580b4d60 2024-03-25T18:34:42.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.229+0000 7fbd557fa640 10 client.4514.objecter resend_mon_ops 2024-03-25T18:34:42.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 789192833 0 0) 0x7fbd3c013b50 con 0x7fbd3c006af0 2024-03-25T18:34:42.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 >> v1:172.21.15.133:6789/0 conn(0x7fbd580b1730 legacy=0x7fbd3c00fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 >> v1:172.21.15.55:6789/0 conn(0x7fbd580b4d60 legacy=0x7fbd3c006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:42.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 1 monclient: found mon.c 2024-03-25T18:34:42.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fbd3c014da0 con 0x7fbd3c006af0 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd3c004eb0 con 0x7fbd3c006af0 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbd440016a0 con 0x7fbd3c006af0 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.236+0000 7fbd557fa640 10 monclient: dump: 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:42.238 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:42.239 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:42.239 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:42.239 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:42.239 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:42.239 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:42.239 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:42.239 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:42.239 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:42.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.560+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd44001800 con 0x7fbd3c006af0 2024-03-25T18:34:42.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.560+0000 7fbd557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:42.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.560+0000 7fbd56ffd640 4 set_mon_vals no callback set 2024-03-25T18:34:42.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.561+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fbd44001ab0 con 0x7fbd3c006af0 2024-03-25T18:34:42.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.561+0000 7fbd5c915640 10 monclient: get_auth_request con 0x7fbd5814be90 auth_method 0 2024-03-25T18:34:42.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.561+0000 7fbd557fa640 10 client.4514.objecter ms_handle_connect 0x7fbd5814be90 2024-03-25T18:34:42.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.567+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fbd44052cc0 con 0x7fbd3c006af0 2024-03-25T18:34:42.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.567+0000 7fbd557fa640 10 client.4514.objecter ms_dispatch 0x7fbd58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:42.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.567+0000 7fbd557fa640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:42.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.567+0000 7fbd557fa640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:42.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.567+0000 7fbd557fa640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-25T18:34:42.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.567+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd4404e7d0 con 0x7fbd3c006af0 2024-03-25T18:34:42.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.567+0000 7fbd557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:42.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.567+0000 7fbd557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:42.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.824+0000 7fbd5eb79640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:42.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.824+0000 7fbd5eb79640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:42.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.824+0000 7fbd5eb79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:34:42.825 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.824+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd580627b0 con 0x7fbd3c006af0 2024-03-25T18:34:42.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.847+0000 7fbd557fa640 1 -- 172.21.15.55:0/4145529667 <== mon.2 v1:172.21.15.55:6790/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fbd44050700 con 0x7fbd3c006af0 2024-03-25T18:34:42.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.847+0000 7fbd557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:42.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.847+0000 7fbd557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:42.848 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:34:42.848 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:42.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 10 librados: watch_flush enter 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 10 librados: watch_flush exit 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 10 client.4514.objecter successfully canceled tick 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 >> v1:172.21.15.55:6800/286536064 conn(0x7fbd5814be90 legacy=0x7fbd3c07bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 10 monclient: shutdown 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 >> v1:172.21.15.55:6790/0 conn(0x7fbd3c006af0 legacy=0x7fbd3c00c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 shutdown_connections 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 >> 172.21.15.55:0/4145529667 conn(0x7fbd580617a0 msgr2=0x7fbd580b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 shutdown_connections 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 1 -- 172.21.15.55:0/4145529667 wait complete. 2024-03-25T18:34:42.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:42.849+0000 7fbd5eb79640 1 librados: shutdown 2024-03-25T18:34:42.862 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:49.669 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:49.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.785+0000 7fc4c0cd6640 10 monclient: get_monmap_and_config 2024-03-25T18:34:49.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.785+0000 7fc4c0cd6640 10 monclient: build_initial_monmap 2024-03-25T18:34:49.786 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.785+0000 7fc4c0cd6640 10 monclient: monmap: 2024-03-25T18:34:49.786 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:49.786 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:49.786 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:34:49.786376+0000 2024-03-25T18:34:49.786 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:34:49.786376+0000 2024-03-25T18:34:49.786 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:49.786 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:49.787 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:49.787 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:34:49.787 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:34:49.787 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:49.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.786+0000 7fc4c0cd6640 1 Processor -- start 2024-03-25T18:34:49.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.786+0000 7fc4c0cd6640 1 -- start start 2024-03-25T18:34:49.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.786+0000 7fc4c0cd6640 10 monclient: init 2024-03-25T18:34:49.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:49.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:49.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 10 monclient(hunting): picked mon.noname-b con 0x7fc4bc0a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:49.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 10 monclient(hunting): picked mon.noname-a con 0x7fc4bc13f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:49.788 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 10 monclient(hunting): picked mon.noname-c con 0x7fc4bc1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:49.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc142520 con 0x7fc4bc13f710 2024-03-25T18:34:49.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0b28a0 con 0x7fc4bc1430c0 2024-03-25T18:34:49.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0b2a30 con 0x7fc4bc0a98f0 2024-03-25T18:34:49.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:49.789 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.787+0000 7fc4c0cd6640 10 monclient(hunting): authenticate will time out at 7694.044931s 2024-03-25T18:34:49.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.791+0000 7fc4bad9d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fc4bc1430c0 0x7fc4bc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:34110/0 (socket says 172.21.15.55:34110) 2024-03-25T18:34:49.792 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.791+0000 7fc4bad9d640 1 -- 172.21.15.55:0/2146529964 learned_addr learned my addr 172.21.15.55:0/2146529964 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:49.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.799+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824374812 0 0) 0x7fc4bc0b28a0 con 0x7fc4bc1430c0 2024-03-25T18:34:49.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.799+0000 7fc4b9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:49.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.799+0000 7fc4b9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:49.800 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.799+0000 7fc4b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:49.801 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.799+0000 7fc4b9d9b640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:34:49.801 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.799+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a8003520 con 0x7fc4bc1430c0 2024-03-25T18:34:49.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.801+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594685109 0 0) 0x7fc4bc142520 con 0x7fc4bc13f710 2024-03-25T18:34:49.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.801+0000 7fc4b9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:49.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.801+0000 7fc4b9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:49.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.801+0000 7fc4b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:49.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.801+0000 7fc4b9d9b640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:34:49.802 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.801+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4bc0b28a0 con 0x7fc4bc13f710 2024-03-25T18:34:49.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1641862062 0 0) 0x7fc4a8003520 con 0x7fc4bc1430c0 2024-03-25T18:34:49.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 >> v1:172.21.15.55:6790/0 conn(0x7fc4bc0a98f0 legacy=0x7fc4bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 >> v1:172.21.15.55:6789/0 conn(0x7fc4bc13f710 legacy=0x7fc4bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:34:49.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 1 monclient: found mon.noname-c 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0b44c0 con 0x7fc4bc1430c0 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fc4b4001a10 con 0x7fc4bc1430c0 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4b9d9b640 10 monclient: dump: 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:49.811 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:49.812 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:49.812 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:49.812 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:49.812 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:49.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4c0cd6640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:34:49.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.809+0000 7fc4c0cd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.813+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b4001c30 con 0x7fc4bc1430c0 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.813+0000 7fc4b9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.813+0000 7fc4b9d9b640 1 -- 172.21.15.55:0/2146529964 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fc4b4001fc0 con 0x7fc4bc1430c0 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.813+0000 7fc4b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.813+0000 7fc4b9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.813+0000 7fc4b9d9b640 10 monclient: dump: 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:49.814 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:49.815 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:49.815 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:49.815 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:49.815 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:49.815 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:49.815 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:49.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.813+0000 7fc4c0cd6640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:49.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.813+0000 7fc4c0cd6640 4 set_mon_vals no callback set 2024-03-25T18:34:49.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.814+0000 7fc4c0cd6640 10 monclient: shutdown 2024-03-25T18:34:49.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.814+0000 7fc4c0cd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.814+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/2146529964 >> v1:172.21.15.133:6789/0 conn(0x7fc4bc1430c0 legacy=0x7fc4bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.814+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/2146529964 shutdown_connections 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.814+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/2146529964 >> 172.21.15.55:0/2146529964 conn(0x7fc4bc0b7ea0 msgr2=0x7fc4bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.814+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/2146529964 shutdown_connections 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.814+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/2146529964 wait complete. 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.815+0000 7fc4c0cd6640 10 monclient: build_initial_monmap 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.815+0000 7fc4c0cd6640 10 monclient: monmap: 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:49.817 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:34:49.818 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:34:49.818 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:49.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.815+0000 7fc4c0cd6640 1 librados: starting msgr at 2024-03-25T18:34:49.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.815+0000 7fc4c0cd6640 1 librados: starting objecter 2024-03-25T18:34:49.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.815+0000 7fc4c0cd6640 1 Processor -- start 2024-03-25T18:34:49.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.815+0000 7fc4c0cd6640 1 -- start start 2024-03-25T18:34:49.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.815+0000 7fc4c0cd6640 1 librados: setting wanted keys 2024-03-25T18:34:49.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.815+0000 7fc4c0cd6640 1 librados: calling monclient init 2024-03-25T18:34:49.818 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.815+0000 7fc4c0cd6640 10 monclient: init 2024-03-25T18:34:49.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:49.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:49.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 10 monclient(hunting): picked mon.noname-a con 0x7fc4bc0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:49.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 10 monclient(hunting): picked mon.noname-c con 0x7fc4bc13f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:49.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 10 monclient(hunting): picked mon.noname-b con 0x7fc4bc1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:49.819 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc137190 con 0x7fc4bc0a98f0 2024-03-25T18:34:49.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc1d41e0 con 0x7fc4bc1430c0 2024-03-25T18:34:49.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc1d4430 con 0x7fc4bc13f710 2024-03-25T18:34:49.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:49.820 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.816+0000 7fc4c0cd6640 10 monclient(hunting): authenticate will time out at 7694.074206s 2024-03-25T18:34:49.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.835+0000 7fc4b3fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fc4bc13f710 0x7fc4bc1352e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:52778/0 (socket says 172.21.15.55:52778) 2024-03-25T18:34:49.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.835+0000 7fc4b3fff640 1 -- 172.21.15.55:0/3186826497 learned_addr learned my addr 172.21.15.55:0/3186826497 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:49.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.835+0000 7fc4b2ffd640 10 client.?.objecter ms_handle_connect 0x7fc4bc13f710 2024-03-25T18:34:49.837 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.835+0000 7fc4b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:49.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.842+0000 7fc4b2ffd640 10 client.?.objecter ms_handle_connect 0x7fc4bc1430c0 2024-03-25T18:34:49.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.842+0000 7fc4b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:49.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.847+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893617107 0 0) 0x7fc4bc1d41e0 con 0x7fc4bc1430c0 2024-03-25T18:34:49.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.847+0000 7fc4b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:49.848 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.847+0000 7fc4b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:49.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.847+0000 7fc4b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:49.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.847+0000 7fc4b2ffd640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:34:49.849 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.847+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c004eb0 con 0x7fc4bc1430c0 2024-03-25T18:34:49.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.849+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989958802 0 0) 0x7fc4bc1d4430 con 0x7fc4bc13f710 2024-03-25T18:34:49.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.849+0000 7fc4b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:49.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.849+0000 7fc4b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:49.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.849+0000 7fc4b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:49.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.849+0000 7fc4b2ffd640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:34:49.850 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.849+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4bc1d41e0 con 0x7fc4bc13f710 2024-03-25T18:34:49.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.850+0000 7fc4b2ffd640 10 client.?.objecter ms_handle_connect 0x7fc4bc0a98f0 2024-03-25T18:34:49.851 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.850+0000 7fc4b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:49.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534572978 0 0) 0x7fc49c004eb0 con 0x7fc4bc1430c0 2024-03-25T18:34:49.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4bc1d4430 con 0x7fc4bc1430c0 2024-03-25T18:34:49.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fc4b4001f40 con 0x7fc4bc1430c0 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 10 monclient(hunting): dump: 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:49.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3519463797 0 0) 0x7fc4bc1d41e0 con 0x7fc4bc13f710 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc49c004eb0 con 0x7fc4bc13f710 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fc4a40019f0 con 0x7fc4bc13f710 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.863+0000 7fc4b2ffd640 10 monclient(hunting): dump: 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:49.866 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 467194892 0 0) 0x7fc4bc1d4430 con 0x7fc4bc1430c0 2024-03-25T18:34:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 >> v1:172.21.15.55:6790/0 conn(0x7fc4bc13f710 legacy=0x7fc4bc1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 >> v1:172.21.15.55:6789/0 conn(0x7fc4bc0a98f0 legacy=0x7fc4bc1d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:49.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 1 monclient: found mon.b 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc13bf90 con 0x7fc4bc1430c0 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4c0cd6640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4c0cd6640 10 monclient: _renew_subs 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4c0cd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc1d5060 con 0x7fc4bc1430c0 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4c0cd6640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4c0cd6640 10 monclient: _renew_subs 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4c0cd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:49.873 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc139f00 con 0x7fc4bc1430c0 2024-03-25T18:34:49.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.871+0000 7fc4c0cd6640 1 librados: init done 2024-03-25T18:34:49.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.872+0000 7fc4c0cd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:49.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.872+0000 7fc4c0cd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:49.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.872+0000 7fc4c0cd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:49.874 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:49.872+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc1d45c0 con 0x7fc4bc1430c0 2024-03-25T18:34:50.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.152+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b4001580 con 0x7fc4bc1430c0 2024-03-25T18:34:50.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.152+0000 7fc4b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:50.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.152+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fc4b4004e10 con 0x7fc4bc1430c0 2024-03-25T18:34:50.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.152+0000 7fc4b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:50.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.152+0000 7fc4b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:50.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.152+0000 7fc4b2ffd640 10 monclient: dump: 2024-03-25T18:34:50.153 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:50.153 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:50.154 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:50.154 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:50.154 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:50.154 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:50.154 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:50.154 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:50.154 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:50.154 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:50.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.152+0000 7fc4b959a640 4 set_mon_vals no callback set 2024-03-25T18:34:50.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.153+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fc4b401ae60 con 0x7fc4bc1430c0 2024-03-25T18:34:50.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.153+0000 7fc4ba59c640 10 monclient: get_auth_request con 0x7fc49c06de90 auth_method 0 2024-03-25T18:34:50.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.154+0000 7fc4b2ffd640 10 client.4510.objecter ms_handle_connect 0x7fc49c06de90 2024-03-25T18:34:50.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.158+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fc4b4054a70 con 0x7fc4bc1430c0 2024-03-25T18:34:50.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.158+0000 7fc4b2ffd640 10 client.4510.objecter ms_dispatch 0x7fc4bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:50.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.158+0000 7fc4b2ffd640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:50.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.158+0000 7fc4b2ffd640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:50.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.158+0000 7fc4b2ffd640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T18:34:50.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.159+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4b40503b0 con 0x7fc4bc1430c0 2024-03-25T18:34:50.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.159+0000 7fc4b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:50.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.159+0000 7fc4b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:50.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.407+0000 7fc4c0cd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:50.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.407+0000 7fc4c0cd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:50.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.407+0000 7fc4c0cd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:50.408 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.407+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4bc0627b0 con 0x7fc4bc1430c0 2024-03-25T18:34:50.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.446+0000 7fc4b2ffd640 1 -- 172.21.15.55:0/3186826497 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fc4b40524b0 con 0x7fc4bc1430c0 2024-03-25T18:34:50.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.446+0000 7fc4b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:50.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.446+0000 7fc4b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:50.447 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:34:50.447 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:50.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 10 librados: watch_flush enter 2024-03-25T18:34:50.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 10 librados: watch_flush exit 2024-03-25T18:34:50.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T18:34:50.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 10 client.4510.objecter successfully canceled tick 2024-03-25T18:34:50.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 >> v1:172.21.15.55:6800/286536064 conn(0x7fc49c06de90 legacy=0x7fc49c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:50.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 10 monclient: shutdown 2024-03-25T18:34:50.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:50.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 >> v1:172.21.15.133:6789/0 conn(0x7fc4bc1430c0 legacy=0x7fc4bc130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:50.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 shutdown_connections 2024-03-25T18:34:50.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.448+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 >> 172.21.15.55:0/3186826497 conn(0x7fc4bc0b7ea0 msgr2=0x7fc4bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:50.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.449+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 shutdown_connections 2024-03-25T18:34:50.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.449+0000 7fc4c0cd6640 1 -- 172.21.15.55:0/3186826497 wait complete. 2024-03-25T18:34:50.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:50.449+0000 7fc4c0cd6640 1 librados: shutdown 2024-03-25T18:34:50.462 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:55.999 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:07:06.169 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:56.003 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:07:06.173 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:56.086 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:07:06.256 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:56.115 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:06.252 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:56.123 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:06.260 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:56.138 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:07:06.308 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:56.145 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:06.283 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:56.146 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:07:06.315 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:56.167 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:07:06.337 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:57.268 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.378+0000 7f1417f58640 10 monclient: get_monmap_and_config 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.378+0000 7f1417f58640 10 monclient: build_initial_monmap 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.378+0000 7f1417f58640 10 monclient: monmap: 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:34:57.378980+0000 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:34:57.378980+0000 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:34:57.379 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.378+0000 7f1417f58640 1 Processor -- start 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 1 -- start start 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 10 monclient: init 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 10 monclient(hunting): picked mon.noname-a con 0x7f14100a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 10 monclient(hunting): picked mon.noname-b con 0x7f141013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 10 monclient(hunting): picked mon.noname-c con 0x7f14101430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1410142520 con 0x7f14100a98f0 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14100b28a0 con 0x7f14101430c0 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14100b2a30 con 0x7f141013f710 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:57.381 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.379+0000 7f1417f58640 10 monclient(hunting): authenticate will time out at 7701.637213s 2024-03-25T18:34:57.388 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.388+0000 7f14164f5640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f14101430c0 0x7f141014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:38728/0 (socket says 172.21.15.55:38728) 2024-03-25T18:34:57.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.388+0000 7f14164f5640 1 -- 172.21.15.55:0/2511911549 learned_addr learned my addr 172.21.15.55:0/2511911549 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:57.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.396+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718478162 0 0) 0x7f14100b28a0 con 0x7f14101430c0 2024-03-25T18:34:57.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.396+0000 7f1414cf2640 10 monclient(hunting): _negotiate 2024-03-25T18:34:57.397 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.396+0000 7f1414cf2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:57.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.396+0000 7f1414cf2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:57.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.396+0000 7f1414cf2640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:34:57.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.396+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13fc003520 con 0x7f14101430c0 2024-03-25T18:34:57.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.397+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674150755 0 0) 0x7f14100b2a30 con 0x7f141013f710 2024-03-25T18:34:57.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.397+0000 7f1414cf2640 10 monclient(hunting): _negotiate 2024-03-25T18:34:57.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.397+0000 7f1414cf2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:57.398 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.397+0000 7f1414cf2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:57.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.397+0000 7f1414cf2640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:34:57.399 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.397+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14100b28a0 con 0x7f141013f710 2024-03-25T18:34:57.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.401+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012692432 0 0) 0x7f1410142520 con 0x7f14100a98f0 2024-03-25T18:34:57.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.401+0000 7f1414cf2640 10 monclient(hunting): _negotiate 2024-03-25T18:34:57.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.401+0000 7f1414cf2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:57.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.401+0000 7f1414cf2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:57.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.401+0000 7f1414cf2640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:34:57.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.401+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14100b2a30 con 0x7f14100a98f0 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1495051701 0 0) 0x7f14100b28a0 con 0x7f141013f710 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 >> v1:172.21.15.133:6789/0 conn(0x7f14101430c0 legacy=0x7f141014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 >> v1:172.21.15.55:6789/0 conn(0x7f14100a98f0 legacy=0x7f14100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 10 monclient: _finish_hunting 0 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 1 monclient: found mon.noname-b 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14100b44c0 con 0x7f141013f710 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 10 monclient: _finish_auth 0 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 10 monclient: _check_auth_tickets 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:57.406 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.404+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1400001670 con 0x7f141013f710 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.405+0000 7f1414cf2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.405+0000 7f1414cf2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.405+0000 7f1414cf2640 10 monclient: dump: 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.405+0000 7f1417f58640 5 monclient: authenticate success, global_id 4523 2024-03-25T18:34:57.407 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.405+0000 7f1417f58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:57.418 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14000017d0 con 0x7f141013f710 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1414cf2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1414cf2640 1 -- 172.21.15.55:0/2511911549 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1400001be0 con 0x7f141013f710 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1417f58640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1417f58640 4 set_mon_vals no callback set 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1414cf2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1414cf2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1414cf2640 10 monclient: dump: 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:57.419 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:57.420 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:57.420 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:57.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1417f58640 10 monclient: shutdown 2024-03-25T18:34:57.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1417f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:57.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1417f58640 1 -- 172.21.15.55:0/2511911549 >> v1:172.21.15.55:6790/0 conn(0x7f141013f710 legacy=0x7f14101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:57.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.418+0000 7f1417f58640 1 -- 172.21.15.55:0/2511911549 shutdown_connections 2024-03-25T18:34:57.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.419+0000 7f1417f58640 1 -- 172.21.15.55:0/2511911549 >> 172.21.15.55:0/2511911549 conn(0x7f14100b7ea0 msgr2=0x7f14100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:57.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.419+0000 7f1417f58640 1 -- 172.21.15.55:0/2511911549 shutdown_connections 2024-03-25T18:34:57.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.419+0000 7f1417f58640 1 -- 172.21.15.55:0/2511911549 wait complete. 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.420+0000 7f1417f58640 10 monclient: build_initial_monmap 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.420+0000 7f1417f58640 10 monclient: monmap: 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.420+0000 7f1417f58640 1 librados: starting msgr at 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.420+0000 7f1417f58640 1 librados: starting objecter 2024-03-25T18:34:57.422 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.420+0000 7f1417f58640 1 Processor -- start 2024-03-25T18:34:57.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.420+0000 7f1417f58640 1 -- start start 2024-03-25T18:34:57.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.420+0000 7f1417f58640 1 librados: setting wanted keys 2024-03-25T18:34:57.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.420+0000 7f1417f58640 1 librados: calling monclient init 2024-03-25T18:34:57.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.420+0000 7f1417f58640 10 monclient: init 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 10 monclient(hunting): picked mon.noname-a con 0x7f14100a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 10 monclient(hunting): picked mon.noname-c con 0x7f141013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 10 monclient(hunting): picked mon.noname-b con 0x7f14101430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14100705b0 con 0x7f14100a98f0 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141006df50 con 0x7f14101430c0 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14101deee0 con 0x7f141013f710 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:57.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.421+0000 7f1417f58640 10 monclient(hunting): authenticate will time out at 7701.678610s 2024-03-25T18:34:57.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.429+0000 7f14164f5640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f14101430c0 0x7f1410139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:38774/0 (socket says 172.21.15.55:38774) 2024-03-25T18:34:57.430 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.429+0000 7f14164f5640 1 -- 172.21.15.55:0/2235002265 learned_addr learned my addr 172.21.15.55:0/2235002265 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:34:57.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.431+0000 7f14067fc640 10 client.?.objecter ms_handle_connect 0x7f14101430c0 2024-03-25T18:34:57.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.431+0000 7f14067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:57.437 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.437+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264919640 0 0) 0x7f141006df50 con 0x7f14101430c0 2024-03-25T18:34:57.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.437+0000 7f14067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:57.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.437+0000 7f14067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:57.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.437+0000 7f14067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:57.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.437+0000 7f14067fc640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:34:57.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.437+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f4004eb0 con 0x7f14101430c0 2024-03-25T18:34:57.438 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.438+0000 7f14067fc640 10 client.?.objecter ms_handle_connect 0x7f141013f710 2024-03-25T18:34:57.439 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.438+0000 7f14067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:57.447 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.447+0000 7f14067fc640 10 client.?.objecter ms_handle_connect 0x7f14100a98f0 2024-03-25T18:34:57.448 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.447+0000 7f14067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:57.450 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.450+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904525470 0 0) 0x7f14101deee0 con 0x7f141013f710 2024-03-25T18:34:57.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.450+0000 7f14067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:57.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.450+0000 7f14067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:57.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.450+0000 7f14067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:57.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.450+0000 7f14067fc640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:34:57.451 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.450+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141006df50 con 0x7f141013f710 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.452+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2293603002 0 0) 0x7f13f4004eb0 con 0x7f14101430c0 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.452+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14101deee0 con 0x7f14101430c0 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.452+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1408001b30 con 0x7f14101430c0 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.452+0000 7f14067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.452+0000 7f14067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.452+0000 7f14067fc640 10 monclient(hunting): dump: 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:57.453 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:57.454 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:57.454 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:57.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.452+0000 7f14067fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:57.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.463+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755208935 0 0) 0x7f14100705b0 con 0x7f14100a98f0 2024-03-25T18:34:57.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.463+0000 7f14067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:57.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.463+0000 7f14067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:57.464 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.463+0000 7f14067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:57.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.463+0000 7f14067fc640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:34:57.465 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.463+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f4004eb0 con 0x7f14100a98f0 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.466+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2580905576 0 0) 0x7f141006df50 con 0x7f141013f710 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.466+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14100705b0 con 0x7f141013f710 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.466+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1400001ae0 con 0x7f141013f710 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.466+0000 7f14067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.466+0000 7f14067fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.466+0000 7f14067fc640 10 monclient(hunting): dump: 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.467 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:57.468 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:57.468 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:57.468 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:57.468 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:57.468 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2670463184 0 0) 0x7f14101deee0 con 0x7f14101430c0 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 >> v1:172.21.15.55:6790/0 conn(0x7f141013f710 legacy=0x7f141006e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 >> v1:172.21.15.55:6789/0 conn(0x7f14100a98f0 legacy=0x7f14101df400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 1 monclient: found mon.b 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14101dfb10 con 0x7f14101430c0 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:57.470 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f14067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f1417f58640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f1417f58640 10 monclient: _renew_subs 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f1417f58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141006f0d0 con 0x7f14101430c0 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f1417f58640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f1417f58640 10 monclient: _renew_subs 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f1417f58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1410139950 con 0x7f14101430c0 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.468+0000 7f1417f58640 1 librados: init done 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.469+0000 7f1417f58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.469+0000 7f1417f58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.469+0000 7f1417f58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:57.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.469+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14101ded40 con 0x7f14101430c0 2024-03-25T18:34:57.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.740+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1408001fe0 con 0x7f14101430c0 2024-03-25T18:34:57.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.740+0000 7f14067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:57.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.740+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1408004b60 con 0x7f14101430c0 2024-03-25T18:34:57.741 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.740+0000 7f14067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.740+0000 7f14067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.740+0000 7f14067fc640 10 monclient: dump: 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:34:57.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.740+0000 7f1407fff640 4 set_mon_vals no callback set 2024-03-25T18:34:57.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.741+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f140801ab30 con 0x7f14101430c0 2024-03-25T18:34:57.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.742+0000 7f1415cf4640 10 monclient: get_auth_request con 0x7f13f406df20 auth_method 0 2024-03-25T18:34:57.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.742+0000 7f14067fc640 10 client.4516.objecter ms_handle_connect 0x7f13f406df20 2024-03-25T18:34:57.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.746+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f1408054900 con 0x7f14101430c0 2024-03-25T18:34:57.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.746+0000 7f14067fc640 10 client.4516.objecter ms_dispatch 0x7f1410000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:57.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.746+0000 7f14067fc640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:57.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.746+0000 7f14067fc640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:57.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.746+0000 7f14067fc640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:34:57.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.747+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1408050240 con 0x7f14101430c0 2024-03-25T18:34:57.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.747+0000 7f14067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:57.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:57.747+0000 7f14067fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:58.004 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.003+0000 7f1417f58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:58.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.004+0000 7f1417f58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:58.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.004+0000 7f1417f58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:34:58.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.004+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14101df070 con 0x7f14101430c0 2024-03-25T18:34:58.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.031+0000 7f14067fc640 1 -- 172.21.15.55:0/2235002265 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f1408052340 con 0x7f14101430c0 2024-03-25T18:34:58.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.031+0000 7f14067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:58.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.031+0000 7f14067fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:58.032 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:34:58.032 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:58.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 10 librados: watch_flush enter 2024-03-25T18:34:58.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 10 librados: watch_flush exit 2024-03-25T18:34:58.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:34:58.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:34:58.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 >> v1:172.21.15.55:6800/286536064 conn(0x7f13f406df20 legacy=0x7f13f40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:58.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 10 monclient: shutdown 2024-03-25T18:34:58.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:58.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 >> v1:172.21.15.133:6789/0 conn(0x7f14101430c0 legacy=0x7f1410139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:58.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 shutdown_connections 2024-03-25T18:34:58.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 >> 172.21.15.55:0/2235002265 conn(0x7f14100b7ea0 msgr2=0x7f1410074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:58.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 shutdown_connections 2024-03-25T18:34:58.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 1 -- 172.21.15.55:0/2235002265 wait complete. 2024-03-25T18:34:58.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:34:58.033+0000 7f1417f58640 1 librados: shutdown 2024-03-25T18:34:58.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:04.852 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:04.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.962+0000 7f7c04b23640 10 monclient: get_monmap_and_config 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.962+0000 7f7c04b23640 10 monclient: build_initial_monmap 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.962+0000 7f7c04b23640 10 monclient: monmap: 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:35:04.963662+0000 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:35:04.963662+0000 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:35:04.964 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:04.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.963+0000 7f7c04b23640 1 Processor -- start 2024-03-25T18:35:04.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.963+0000 7f7c04b23640 1 -- start start 2024-03-25T18:35:04.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.963+0000 7f7c04b23640 10 monclient: init 2024-03-25T18:35:04.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:04.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:04.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 10 monclient(hunting): picked mon.noname-c con 0x7f7c000a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:04.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 10 monclient(hunting): picked mon.noname-b con 0x7f7c0013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:04.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 10 monclient(hunting): picked mon.noname-a con 0x7f7c001430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:04.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c00142520 con 0x7f7c001430c0 2024-03-25T18:35:04.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000b28a0 con 0x7f7c000a98f0 2024-03-25T18:35:04.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000b2a30 con 0x7f7c0013f710 2024-03-25T18:35:04.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:04.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.964+0000 7f7c04b23640 10 monclient(hunting): authenticate will time out at 7709.222128s 2024-03-25T18:35:04.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.978+0000 7f7bfed9d640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f7c001430c0 0x7f7c0014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54194/0 (socket says 172.21.15.55:54194) 2024-03-25T18:35:04.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.978+0000 7f7bfed9d640 1 -- 172.21.15.55:0/1441233180 learned_addr learned my addr 172.21.15.55:0/1441233180 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:04.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.994+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725592469 0 0) 0x7f7c000b28a0 con 0x7f7c000a98f0 2024-03-25T18:35:04.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.994+0000 7f7bfd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:04.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.994+0000 7f7bfd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:04.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.994+0000 7f7bfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:04.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.994+0000 7f7bfd59a640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:35:04.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:04.994+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be8003520 con 0x7f7c000a98f0 2024-03-25T18:35:05.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.005+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467944081 0 0) 0x7f7c00142520 con 0x7f7c001430c0 2024-03-25T18:35:05.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.005+0000 7f7bfd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:05.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.005+0000 7f7bfd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:05.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.005+0000 7f7bfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:05.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.005+0000 7f7bfd59a640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:35:05.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.005+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c000b28a0 con 0x7f7c001430c0 2024-03-25T18:35:05.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.006+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023308698 0 0) 0x7f7c000b2a30 con 0x7f7c0013f710 2024-03-25T18:35:05.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.006+0000 7f7bfd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:05.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.006+0000 7f7bfd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:05.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.006+0000 7f7bfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:05.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.006+0000 7f7bfd59a640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:35:05.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.006+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c00142520 con 0x7f7c0013f710 2024-03-25T18:35:05.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 396166746 0 0) 0x7f7be8003520 con 0x7f7c000a98f0 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 >> v1:172.21.15.55:6790/0 conn(0x7f7c0013f710 legacy=0x7f7c001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 >> v1:172.21.15.55:6789/0 conn(0x7f7c001430c0 legacy=0x7f7c0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 1 monclient: found mon.noname-c 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c000b44c0 con 0x7f7c000a98f0 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 10 monclient: _finish_auth 0 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7bf0001690 con 0x7f7c000a98f0 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:05.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7bfd59a640 10 monclient: dump: 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7c04b23640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:35:05.024 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.021+0000 7f7c04b23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:05.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.046+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bf00017f0 con 0x7f7c000a98f0 2024-03-25T18:35:05.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.046+0000 7f7bfd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:05.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.046+0000 7f7bfd59a640 1 -- 172.21.15.55:0/1441233180 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7bf0001c00 con 0x7f7c000a98f0 2024-03-25T18:35:05.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.046+0000 7f7c04b23640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:05.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.046+0000 7f7c04b23640 4 set_mon_vals no callback set 2024-03-25T18:35:05.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.046+0000 7f7bfd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:05.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.046+0000 7f7bfd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.046+0000 7f7bfd59a640 10 monclient: dump: 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:05.048 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:05.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.047+0000 7f7c04b23640 10 monclient: shutdown 2024-03-25T18:35:05.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.047+0000 7f7c04b23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:05.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.047+0000 7f7c04b23640 1 -- 172.21.15.55:0/1441233180 >> v1:172.21.15.133:6789/0 conn(0x7f7c000a98f0 legacy=0x7f7c000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:05.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.047+0000 7f7c04b23640 1 -- 172.21.15.55:0/1441233180 shutdown_connections 2024-03-25T18:35:05.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.047+0000 7f7c04b23640 1 -- 172.21.15.55:0/1441233180 >> 172.21.15.55:0/1441233180 conn(0x7f7c000b7ea0 msgr2=0x7f7c000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:05.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.047+0000 7f7c04b23640 1 -- 172.21.15.55:0/1441233180 shutdown_connections 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.047+0000 7f7c04b23640 1 -- 172.21.15.55:0/1441233180 wait complete. 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.048+0000 7f7c04b23640 10 monclient: build_initial_monmap 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.048+0000 7f7c04b23640 10 monclient: monmap: 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:05.051 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.048+0000 7f7c04b23640 1 librados: starting msgr at 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.048+0000 7f7c04b23640 1 librados: starting objecter 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.048+0000 7f7c04b23640 1 Processor -- start 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.048+0000 7f7c04b23640 1 -- start start 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.048+0000 7f7c04b23640 1 librados: setting wanted keys 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.048+0000 7f7c04b23640 1 librados: calling monclient init 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.048+0000 7f7c04b23640 10 monclient: init 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 10 monclient(hunting): picked mon.noname-b con 0x7f7c000a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 10 monclient(hunting): picked mon.noname-a con 0x7f7c0013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 10 monclient(hunting): picked mon.noname-c con 0x7f7c001430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c001d76a0 con 0x7f7c0013f710 2024-03-25T18:35:05.052 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c001d82b0 con 0x7f7c000a98f0 2024-03-25T18:35:05.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c001d8440 con 0x7f7c001430c0 2024-03-25T18:35:05.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:05.053 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.049+0000 7f7c04b23640 10 monclient(hunting): authenticate will time out at 7709.306762s 2024-03-25T18:35:05.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.053+0000 7f7bfe59c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f7c000a98f0 0x7f7c001d7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:46252/0 (socket says 172.21.15.55:46252) 2024-03-25T18:35:05.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.053+0000 7f7bfe59c640 1 -- 172.21.15.55:0/1242860878 learned_addr learned my addr 172.21.15.55:0/1242860878 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:05.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.060+0000 7f7beeffd640 10 client.?.objecter ms_handle_connect 0x7f7c000a98f0 2024-03-25T18:35:05.061 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.060+0000 7f7beeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:05.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.064+0000 7f7beeffd640 10 client.?.objecter ms_handle_connect 0x7f7c001430c0 2024-03-25T18:35:05.065 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.064+0000 7f7beeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:05.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.065+0000 7f7beeffd640 10 client.?.objecter ms_handle_connect 0x7f7c0013f710 2024-03-25T18:35:05.066 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.065+0000 7f7beeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:05.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.067+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714411326 0 0) 0x7f7c001d82b0 con 0x7f7c000a98f0 2024-03-25T18:35:05.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.067+0000 7f7beeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:05.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.067+0000 7f7beeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:05.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.067+0000 7f7beeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:05.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.067+0000 7f7beeffd640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:35:05.068 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.067+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be0004eb0 con 0x7f7c000a98f0 2024-03-25T18:35:05.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.074+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829603847 0 0) 0x7f7c001d76a0 con 0x7f7c0013f710 2024-03-25T18:35:05.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.074+0000 7f7beeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:05.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.074+0000 7f7beeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:05.075 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.074+0000 7f7beeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:05.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.074+0000 7f7beeffd640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:35:05.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.074+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c001d82b0 con 0x7f7c0013f710 2024-03-25T18:35:05.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191342134 0 0) 0x7f7c001d8440 con 0x7f7c001430c0 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c001d76a0 con 0x7f7c001430c0 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 872959745 0 0) 0x7f7be0004eb0 con 0x7f7c000a98f0 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c001d8440 con 0x7f7c000a98f0 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7bf0001bc0 con 0x7f7c000a98f0 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 10 monclient(hunting): dump: 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:05.079 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:05.080 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:05.080 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:05.080 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:05.080 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:05.080 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:05.080 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:05.080 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:05.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.077+0000 7f7beeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:05.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2185743221 0 0) 0x7f7c001d8440 con 0x7f7c000a98f0 2024-03-25T18:35:05.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 >> v1:172.21.15.55:6790/0 conn(0x7f7c001430c0 legacy=0x7f7c001e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:05.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 >> v1:172.21.15.55:6789/0 conn(0x7f7c0013f710 legacy=0x7f7c001386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:05.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 1 monclient: found mon.b 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c00138de0 con 0x7f7c000a98f0 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7beeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7c04b23640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7c04b23640 10 monclient: _renew_subs 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7c04b23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c00137f30 con 0x7f7c000a98f0 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7c04b23640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7c04b23640 10 monclient: _renew_subs 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7c04b23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:05.089 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c00135340 con 0x7f7c000a98f0 2024-03-25T18:35:05.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.087+0000 7f7c04b23640 1 librados: init done 2024-03-25T18:35:05.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.088+0000 7f7c04b23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:05.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.088+0000 7f7c04b23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:05.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.088+0000 7f7c04b23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:05.090 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.088+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c00139030 con 0x7f7c000a98f0 2024-03-25T18:35:05.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.352+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bf0001220 con 0x7f7c000a98f0 2024-03-25T18:35:05.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.352+0000 7f7beeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:05.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.352+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7bf0003b00 con 0x7f7c000a98f0 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.352+0000 7f7beeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.352+0000 7f7beeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.352+0000 7f7beeffd640 10 monclient: dump: 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:05.354 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:05.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.352+0000 7f7bfcd99640 4 set_mon_vals no callback set 2024-03-25T18:35:05.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.353+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f7bf0003d20 con 0x7f7c000a98f0 2024-03-25T18:35:05.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.354+0000 7f7bfdd9b640 10 monclient: get_auth_request con 0x7f7be006ddf0 auth_method 0 2024-03-25T18:35:05.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.354+0000 7f7beeffd640 10 client.4522.objecter ms_handle_connect 0x7f7be006ddf0 2024-03-25T18:35:05.358 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.357+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f7bf00548e0 con 0x7f7c000a98f0 2024-03-25T18:35:05.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.357+0000 7f7beeffd640 10 client.4522.objecter ms_dispatch 0x7f7c00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:05.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.357+0000 7f7beeffd640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:05.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.357+0000 7f7beeffd640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:05.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.357+0000 7f7beeffd640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T18:35:05.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.358+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7bf00503f0 con 0x7f7c000a98f0 2024-03-25T18:35:05.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.358+0000 7f7beeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:05.359 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.358+0000 7f7beeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:05.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.598+0000 7f7c04b23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:05.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.598+0000 7f7c04b23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:05.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.598+0000 7f7c04b23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:05.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.598+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c000abde0 con 0x7f7c000a98f0 2024-03-25T18:35:05.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.692+0000 7f7beeffd640 1 -- 172.21.15.55:0/1242860878 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f7bf0052320 con 0x7f7c000a98f0 2024-03-25T18:35:05.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.692+0000 7f7beeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:05.694 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.692+0000 7f7beeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:05.694 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:35:05.694 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:05.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.694+0000 7f7c04b23640 10 librados: watch_flush enter 2024-03-25T18:35:05.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 10 librados: watch_flush exit 2024-03-25T18:35:05.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T18:35:05.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 10 client.4522.objecter successfully canceled tick 2024-03-25T18:35:05.696 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 >> v1:172.21.15.55:6800/286536064 conn(0x7f7be006ddf0 legacy=0x7f7be00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:05.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 10 monclient: shutdown 2024-03-25T18:35:05.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:05.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 >> v1:172.21.15.133:6789/0 conn(0x7f7c000a98f0 legacy=0x7f7c001d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:05.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 shutdown_connections 2024-03-25T18:35:05.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 >> 172.21.15.55:0/1242860878 conn(0x7f7c000b7ea0 msgr2=0x7f7c00074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:05.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 shutdown_connections 2024-03-25T18:35:05.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 1 -- 172.21.15.55:0/1242860878 wait complete. 2024-03-25T18:35:05.697 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:05.695+0000 7f7c04b23640 1 librados: shutdown 2024-03-25T18:35:05.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:12.514 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:12.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.633+0000 7f0a84ac0640 10 monclient: get_monmap_and_config 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.633+0000 7f0a84ac0640 10 monclient: build_initial_monmap 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.633+0000 7f0a84ac0640 10 monclient: monmap: 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:35:12.635767+0000 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:35:12.635767+0000 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:35:12.636 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:35:12.637 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.634+0000 7f0a84ac0640 1 Processor -- start 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.634+0000 7f0a84ac0640 1 -- start start 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.634+0000 7f0a84ac0640 10 monclient: init 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 10 monclient(hunting): picked mon.noname-a con 0x7f0a800b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 10 monclient(hunting): picked mon.noname-c con 0x7f0a800b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 10 monclient(hunting): picked mon.noname-b con 0x7f0a8014be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a800af940 con 0x7f0a800b4d60 2024-03-25T18:35:12.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a800b0a70 con 0x7f0a800b1730 2024-03-25T18:35:12.639 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a800b0d50 con 0x7f0a8014be90 2024-03-25T18:35:12.639 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:12.639 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.635+0000 7f0a84ac0640 10 monclient(hunting): authenticate will time out at 7716.894251s 2024-03-25T18:35:12.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.647+0000 7f0a7dd9b640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f0a800b1730 0x7f0a800b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:48272/0 (socket says 172.21.15.55:48272) 2024-03-25T18:35:12.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.648+0000 7f0a7dd9b640 1 -- 172.21.15.55:0/1356571953 learned_addr learned my addr 172.21.15.55:0/1356571953 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:12.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.660+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162908967 0 0) 0x7f0a800b0a70 con 0x7f0a800b1730 2024-03-25T18:35:12.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.660+0000 7f0a7d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.660+0000 7f0a7d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.660+0000 7f0a7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.660+0000 7f0a7d59a640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:35:12.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.660+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a68003520 con 0x7f0a800b1730 2024-03-25T18:35:12.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.665+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858645026 0 0) 0x7f0a800af940 con 0x7f0a800b4d60 2024-03-25T18:35:12.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.665+0000 7f0a7d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.665+0000 7f0a7d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.665+0000 7f0a7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.665+0000 7f0a7d59a640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:35:12.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.665+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a800b0a70 con 0x7f0a800b4d60 2024-03-25T18:35:12.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.667+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403827356 0 0) 0x7f0a800b0d50 con 0x7f0a8014be90 2024-03-25T18:35:12.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.667+0000 7f0a7d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.667+0000 7f0a7d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.667+0000 7f0a7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.667+0000 7f0a7d59a640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:35:12.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.667+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a800af940 con 0x7f0a8014be90 2024-03-25T18:35:12.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3973752497 0 0) 0x7f0a68003520 con 0x7f0a800b1730 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 >> v1:172.21.15.55:6790/0 conn(0x7f0a8014be90 legacy=0x7f0a8014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 >> v1:172.21.15.55:6789/0 conn(0x7f0a800b4d60 legacy=0x7f0a8013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 1 monclient: found mon.noname-c 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a800b1030 con 0x7f0a800b1730 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 10 monclient: _finish_auth 0 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0a74001690 con 0x7f0a800b1730 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:12.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a7d59a640 10 monclient: dump: 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a84ac0640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:35:12.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.679+0000 7f0a84ac0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:12.707 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.705+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a740017f0 con 0x7f0a800b1730 2024-03-25T18:35:12.707 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.705+0000 7f0a7d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:12.707 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.705+0000 7f0a7d59a640 1 -- 172.21.15.55:0/1356571953 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0a74001c00 con 0x7f0a800b1730 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.705+0000 7f0a84ac0640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.705+0000 7f0a84ac0640 4 set_mon_vals no callback set 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.705+0000 7f0a7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.705+0000 7f0a7d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.705+0000 7f0a7d59a640 10 monclient: dump: 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:12.708 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:12.709 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.706+0000 7f0a84ac0640 10 monclient: shutdown 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.706+0000 7f0a84ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.706+0000 7f0a84ac0640 1 -- 172.21.15.55:0/1356571953 >> v1:172.21.15.133:6789/0 conn(0x7f0a800b1730 legacy=0x7f0a800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.706+0000 7f0a84ac0640 1 -- 172.21.15.55:0/1356571953 shutdown_connections 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.706+0000 7f0a84ac0640 1 -- 172.21.15.55:0/1356571953 >> 172.21.15.55:0/1356571953 conn(0x7f0a800617a0 msgr2=0x7f0a80061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.706+0000 7f0a84ac0640 1 -- 172.21.15.55:0/1356571953 shutdown_connections 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.706+0000 7f0a84ac0640 1 -- 172.21.15.55:0/1356571953 wait complete. 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.707+0000 7f0a84ac0640 10 monclient: build_initial_monmap 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.707+0000 7f0a84ac0640 10 monclient: monmap: 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:35:12.711 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.707+0000 7f0a84ac0640 1 librados: starting msgr at 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.707+0000 7f0a84ac0640 1 librados: starting objecter 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 1 Processor -- start 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 1 -- start start 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 1 librados: setting wanted keys 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 1 librados: calling monclient init 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 10 monclient: init 2024-03-25T18:35:12.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:12.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:12.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 10 monclient(hunting): picked mon.noname-a con 0x7f0a800b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:12.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 10 monclient(hunting): picked mon.noname-b con 0x7f0a800b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:12.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 10 monclient(hunting): picked mon.noname-c con 0x7f0a8014be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:12.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a801d77a0 con 0x7f0a800b1730 2024-03-25T18:35:12.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a80143090 con 0x7f0a800b4d60 2024-03-25T18:35:12.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a80143220 con 0x7f0a8014be90 2024-03-25T18:35:12.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:12.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.708+0000 7f0a84ac0640 10 monclient(hunting): authenticate will time out at 7716.967208s 2024-03-25T18:35:12.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.721+0000 7f0a7dd9b640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f0a800b4d60 0x7f0a801d8060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:48340/0 (socket says 172.21.15.55:48340) 2024-03-25T18:35:12.723 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.721+0000 7f0a7dd9b640 1 -- 172.21.15.55:0/2921639708 learned_addr learned my addr 172.21.15.55:0/2921639708 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:12.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.723+0000 7f0a6effd640 10 client.?.objecter ms_handle_connect 0x7f0a800b4d60 2024-03-25T18:35:12.725 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.723+0000 7f0a6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:12.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.730+0000 7f0a6effd640 10 client.?.objecter ms_handle_connect 0x7f0a8014be90 2024-03-25T18:35:12.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.730+0000 7f0a6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:12.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.733+0000 7f0a6effd640 10 client.?.objecter ms_handle_connect 0x7f0a800b1730 2024-03-25T18:35:12.735 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.733+0000 7f0a6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:12.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.740+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561248346 0 0) 0x7f0a80143090 con 0x7f0a800b4d60 2024-03-25T18:35:12.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.740+0000 7f0a6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.740+0000 7f0a6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.740+0000 7f0a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.740+0000 7f0a6effd640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:35:12.742 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.740+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a60004eb0 con 0x7f0a800b4d60 2024-03-25T18:35:12.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.747+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139292678 0 0) 0x7f0a801d77a0 con 0x7f0a800b1730 2024-03-25T18:35:12.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.747+0000 7f0a6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.747+0000 7f0a6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.747+0000 7f0a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.747+0000 7f0a6effd640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:35:12.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.747+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a80143090 con 0x7f0a800b1730 2024-03-25T18:35:12.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.749+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422910440 0 0) 0x7f0a80143220 con 0x7f0a8014be90 2024-03-25T18:35:12.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.749+0000 7f0a6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:12.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.749+0000 7f0a6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:12.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.749+0000 7f0a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:12.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.749+0000 7f0a6effd640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:35:12.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.749+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a801d77a0 con 0x7f0a8014be90 2024-03-25T18:35:12.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.756+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3016165446 0 0) 0x7f0a80143090 con 0x7f0a800b1730 2024-03-25T18:35:12.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.756+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a80143220 con 0x7f0a800b1730 2024-03-25T18:35:12.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.756+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0a70001970 con 0x7f0a800b1730 2024-03-25T18:35:12.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.756+0000 7f0a6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:12.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.756+0000 7f0a6effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:12.758 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.756+0000 7f0a6effd640 10 monclient(hunting): dump: 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:12.759 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.756+0000 7f0a6effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:12.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.761+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1540424631 0 0) 0x7f0a60004eb0 con 0x7f0a800b4d60 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.761+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a80143090 con 0x7f0a800b4d60 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.762+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0a74001bc0 con 0x7f0a800b4d60 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.762+0000 7f0a6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.762+0000 7f0a6effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.762+0000 7f0a6effd640 10 monclient(hunting): dump: 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:12.764 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:12.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.763+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1529447067 0 0) 0x7f0a80143220 con 0x7f0a800b1730 2024-03-25T18:35:12.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.763+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 >> v1:172.21.15.55:6790/0 conn(0x7f0a8014be90 legacy=0x7f0a8013b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.763+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 >> v1:172.21.15.133:6789/0 conn(0x7f0a800b4d60 legacy=0x7f0a801d8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:12.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.763+0000 7f0a6effd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:12.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.763+0000 7f0a6effd640 1 monclient: found mon.a 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.763+0000 7f0a6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.763+0000 7f0a6effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.763+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a801433b0 con 0x7f0a800b1730 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a6effd640 10 monclient: _finish_auth 0 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a6effd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a84ac0640 5 monclient: authenticate success, global_id 4551 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a84ac0640 10 monclient: _renew_subs 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a84ac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8013bd20 con 0x7f0a800b1730 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a84ac0640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a84ac0640 10 monclient: _renew_subs 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a84ac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a801428b0 con 0x7f0a800b1730 2024-03-25T18:35:12.767 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.764+0000 7f0a84ac0640 1 librados: init done 2024-03-25T18:35:12.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.765+0000 7f0a84ac0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:12.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.765+0000 7f0a84ac0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:12.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.765+0000 7f0a84ac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:12.768 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:12.765+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a801d7eb0 con 0x7f0a800b1730 2024-03-25T18:35:13.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.083+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a70003610 con 0x7f0a800b1730 2024-03-25T18:35:13.085 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.083+0000 7f0a6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.083+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0a70003920 con 0x7f0a800b1730 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.083+0000 7f0a6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.083+0000 7f0a6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.083+0000 7f0a6effd640 10 monclient: dump: 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:13.086 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:13.087 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:13.087 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:13.087 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:13.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.083+0000 7f0a7cd99640 4 set_mon_vals no callback set 2024-03-25T18:35:13.087 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.084+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f0a700198f0 con 0x7f0a800b1730 2024-03-25T18:35:13.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.084+0000 7f0a7dd9b640 10 monclient: get_auth_request con 0x7f0a6006df20 auth_method 0 2024-03-25T18:35:13.088 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.085+0000 7f0a6effd640 10 client.4551.objecter ms_handle_connect 0x7f0a6006df20 2024-03-25T18:35:13.093 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.091+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f0a700535e0 con 0x7f0a800b1730 2024-03-25T18:35:13.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.091+0000 7f0a6effd640 10 client.4551.objecter ms_dispatch 0x7f0a80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:13.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.091+0000 7f0a6effd640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:13.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.091+0000 7f0a6effd640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:13.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.091+0000 7f0a6effd640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T18:35:13.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.091+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a7004ef20 con 0x7f0a800b1730 2024-03-25T18:35:13.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.091+0000 7f0a6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:13.094 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.091+0000 7f0a6effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:13.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.328+0000 7f0a84ac0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:13.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.328+0000 7f0a84ac0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:13.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.328+0000 7f0a84ac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:13.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.328+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a801d7990 con 0x7f0a800b1730 2024-03-25T18:35:13.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.351+0000 7f0a6effd640 1 -- 172.21.15.55:0/2921639708 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f0a70051020 con 0x7f0a800b1730 2024-03-25T18:35:13.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.351+0000 7f0a6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:13.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.351+0000 7f0a6effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:13.353 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:35:13.353 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:13.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 10 librados: watch_flush enter 2024-03-25T18:35:13.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 10 librados: watch_flush exit 2024-03-25T18:35:13.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T18:35:13.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 10 client.4551.objecter successfully canceled tick 2024-03-25T18:35:13.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 >> v1:172.21.15.55:6800/286536064 conn(0x7f0a6006df20 legacy=0x7f0a600703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:13.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 10 monclient: shutdown 2024-03-25T18:35:13.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:13.356 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 >> v1:172.21.15.55:6789/0 conn(0x7f0a800b1730 legacy=0x7f0a8013ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:13.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 shutdown_connections 2024-03-25T18:35:13.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.353+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 >> 172.21.15.55:0/2921639708 conn(0x7f0a800617a0 msgr2=0x7f0a800b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:13.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.354+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 shutdown_connections 2024-03-25T18:35:13.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.354+0000 7f0a84ac0640 1 -- 172.21.15.55:0/2921639708 wait complete. 2024-03-25T18:35:13.357 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:13.354+0000 7f0a84ac0640 1 librados: shutdown 2024-03-25T18:35:13.369 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:16.093 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:07:26.263 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:16.100 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:07:26.270 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:16.132 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:07:26.302 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:16.153 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:26.290 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:16.158 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:26.295 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:16.162 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:07:26.332 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:16.169 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:07:26.339 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:16.194 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:26.331 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:16.198 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:07:26.368 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:20.175 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:20.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.293+0000 7f1be05d6640 10 monclient: get_monmap_and_config 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.293+0000 7f1be05d6640 10 monclient: build_initial_monmap 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.293+0000 7f1be05d6640 10 monclient: monmap: 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:35:20.294188+0000 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:35:20.294188+0000 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:35:20.295 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:20.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.293+0000 7f1be05d6640 1 Processor -- start 2024-03-25T18:35:20.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 1 -- start start 2024-03-25T18:35:20.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 10 monclient: init 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd80a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 10 monclient(hunting): picked mon.noname-b con 0x7f1bd81430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd8142520 con 0x7f1bd80a98f0 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd80b28a0 con 0x7f1bd813f710 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd80b2a30 con 0x7f1bd81430c0 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:20.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.294+0000 7f1be05d6640 10 monclient(hunting): authenticate will time out at 7724.552314s 2024-03-25T18:35:20.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.305+0000 7f1bde372640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1bd80a98f0 0x7f1bd80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:43174/0 (socket says 172.21.15.55:43174) 2024-03-25T18:35:20.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.305+0000 7f1bde372640 1 -- 172.21.15.55:0/832137615 learned_addr learned my addr 172.21.15.55:0/832137615 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:20.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.313+0000 7f1bdd370640 1 -- 172.21.15.55:0/832137615 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636275285 0 0) 0x7f1bd8142520 con 0x7f1bd80a98f0 2024-03-25T18:35:20.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.313+0000 7f1bdd370640 10 monclient(hunting): _negotiate 2024-03-25T18:35:20.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.313+0000 7f1bdd370640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.313+0000 7f1bdd370640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.313+0000 7f1bdd370640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:35:20.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.313+0000 7f1bdd370640 1 -- 172.21.15.55:0/832137615 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc4003520 con 0x7f1bd80a98f0 2024-03-25T18:35:20.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 1 -- 172.21.15.55:0/832137615 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1137782766 0 0) 0x7f1bc4003520 con 0x7f1bd80a98f0 2024-03-25T18:35:20.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 1 -- 172.21.15.55:0/832137615 >> v1:172.21.15.55:6790/0 conn(0x7f1bd81430c0 legacy=0x7f1bd814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 1 -- 172.21.15.55:0/832137615 >> v1:172.21.15.133:6789/0 conn(0x7f1bd813f710 legacy=0x7f1bd81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 10 monclient: _finish_hunting 0 2024-03-25T18:35:20.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 1 monclient: found mon.noname-a 2024-03-25T18:35:20.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:20.327 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 1 -- 172.21.15.55:0/832137615 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd80b44c0 con 0x7f1bd80a98f0 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 10 monclient: _finish_auth 0 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 10 monclient: _check_auth_tickets 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 1 -- 172.21.15.55:0/832137615 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1bd4001690 con 0x7f1bd80a98f0 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1bdd370640 10 monclient: dump: 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:20.328 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:20.329 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:20.329 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:20.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1be05d6640 5 monclient: authenticate success, global_id 4554 2024-03-25T18:35:20.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.326+0000 7f1be05d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:20.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.332+0000 7f1bdd370640 1 -- 172.21.15.55:0/832137615 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bd40017f0 con 0x7f1bd80a98f0 2024-03-25T18:35:20.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.332+0000 7f1bdd370640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:20.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.332+0000 7f1bdd370640 1 -- 172.21.15.55:0/832137615 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1bd4001c00 con 0x7f1bd80a98f0 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.332+0000 7f1bdd370640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.332+0000 7f1bdd370640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.332+0000 7f1bdd370640 10 monclient: dump: 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:20.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.332+0000 7f1be05d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:20.335 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.332+0000 7f1be05d6640 4 set_mon_vals no callback set 2024-03-25T18:35:20.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.333+0000 7f1be05d6640 10 monclient: shutdown 2024-03-25T18:35:20.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.333+0000 7f1be05d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.333+0000 7f1be05d6640 1 -- 172.21.15.55:0/832137615 >> v1:172.21.15.55:6789/0 conn(0x7f1bd80a98f0 legacy=0x7f1bd80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.333+0000 7f1be05d6640 1 -- 172.21.15.55:0/832137615 shutdown_connections 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.333+0000 7f1be05d6640 1 -- 172.21.15.55:0/832137615 >> 172.21.15.55:0/832137615 conn(0x7f1bd80b7ea0 msgr2=0x7f1bd80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.333+0000 7f1be05d6640 1 -- 172.21.15.55:0/832137615 shutdown_connections 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.333+0000 7f1be05d6640 1 -- 172.21.15.55:0/832137615 wait complete. 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.334+0000 7f1be05d6640 10 monclient: build_initial_monmap 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.334+0000 7f1be05d6640 10 monclient: monmap: 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:20.337 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.334+0000 7f1be05d6640 1 librados: starting msgr at 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.334+0000 7f1be05d6640 1 librados: starting objecter 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.334+0000 7f1be05d6640 1 Processor -- start 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 1 -- start start 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 1 librados: setting wanted keys 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 1 librados: calling monclient init 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 10 monclient: init 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd80a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 10 monclient(hunting): picked mon.noname-b con 0x7f1bd813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:20.338 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd81430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:20.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd8138c70 con 0x7f1bd81430c0 2024-03-25T18:35:20.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd81367d0 con 0x7f1bd813f710 2024-03-25T18:35:20.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd8136960 con 0x7f1bd80a98f0 2024-03-25T18:35:20.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:20.339 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.335+0000 7f1be05d6640 10 monclient(hunting): authenticate will time out at 7724.593238s 2024-03-25T18:35:20.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.341+0000 7f1bdeb73640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f1bd81430c0 0x7f1bd81e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:43220/0 (socket says 172.21.15.55:43220) 2024-03-25T18:35:20.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.341+0000 7f1bdeb73640 1 -- 172.21.15.55:0/2440890678 learned_addr learned my addr 172.21.15.55:0/2440890678 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:20.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.342+0000 7f1bceffd640 10 client.?.objecter ms_handle_connect 0x7f1bd81430c0 2024-03-25T18:35:20.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.342+0000 7f1bceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:20.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.346+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294271817 0 0) 0x7f1bd8138c70 con 0x7f1bd81430c0 2024-03-25T18:35:20.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.346+0000 7f1bceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:20.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.346+0000 7f1bceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:20.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.346+0000 7f1bceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:20.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.346+0000 7f1bceffd640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:35:20.347 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.346+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc004eb0 con 0x7f1bd81430c0 2024-03-25T18:35:20.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.353+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4152051113 0 0) 0x7f1bbc004eb0 con 0x7f1bd81430c0 2024-03-25T18:35:20.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.353+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bd8138c70 con 0x7f1bd81430c0 2024-03-25T18:35:20.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.353+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1bd0001c90 con 0x7f1bd81430c0 2024-03-25T18:35:20.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.353+0000 7f1bceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:20.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.353+0000 7f1bceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:20.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.353+0000 7f1bceffd640 10 monclient(hunting): dump: 2024-03-25T18:35:20.354 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:20.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.353+0000 7f1bceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:20.358 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.357+0000 7f1bceffd640 10 client.?.objecter ms_handle_connect 0x7f1bd80a98f0 2024-03-25T18:35:20.358 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.357+0000 7f1bceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:20.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3077091507 0 0) 0x7f1bd8138c70 con 0x7f1bd81430c0 2024-03-25T18:35:20.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 >> v1:172.21.15.55:6790/0 conn(0x7f1bd80a98f0 legacy=0x7f1bd8137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 >> v1:172.21.15.133:6789/0 conn(0x7f1bd813f710 legacy=0x7f1bd8132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:20.361 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 1 monclient: found mon.a 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd8139630 con 0x7f1bd81430c0 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1bceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1be05d6640 5 monclient: authenticate success, global_id 4557 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1be05d6640 10 monclient: _renew_subs 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1be05d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd8139880 con 0x7f1bd81430c0 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1be05d6640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1be05d6640 10 monclient: _renew_subs 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1be05d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:20.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd81337f0 con 0x7f1bd81430c0 2024-03-25T18:35:20.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.360+0000 7f1be05d6640 1 librados: init done 2024-03-25T18:35:20.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.361+0000 7f1be05d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:20.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.361+0000 7f1be05d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:20.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.361+0000 7f1be05d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:20.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.361+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd80627b0 con 0x7f1bd81430c0 2024-03-25T18:35:20.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.678+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bd0003990 con 0x7f1bd81430c0 2024-03-25T18:35:20.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.678+0000 7f1bceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:20.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.678+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f1bd0003da0 con 0x7f1bd81430c0 2024-03-25T18:35:20.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.678+0000 7f1bceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:20.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.678+0000 7f1bceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:20.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.678+0000 7f1bceffd640 10 monclient: dump: 2024-03-25T18:35:20.680 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:20.680 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:20.680 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.678+0000 7f1bdcb6f640 4 set_mon_vals no callback set 2024-03-25T18:35:20.681 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.679+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f1bd0003f80 con 0x7f1bd81430c0 2024-03-25T18:35:20.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.680+0000 7f1bde372640 10 monclient: get_auth_request con 0x7f1bbc06dbf0 auth_method 0 2024-03-25T18:35:20.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.680+0000 7f1bceffd640 10 client.4557.objecter ms_handle_connect 0x7f1bbc06dbf0 2024-03-25T18:35:20.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.686+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f1bd0054c30 con 0x7f1bd81430c0 2024-03-25T18:35:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.686+0000 7f1bceffd640 10 client.4557.objecter ms_dispatch 0x7f1bd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.686+0000 7f1bceffd640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.686+0000 7f1bceffd640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.686+0000 7f1bceffd640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-25T18:35:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.686+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bd0050740 con 0x7f1bd81430c0 2024-03-25T18:35:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.686+0000 7f1bceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:20.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.686+0000 7f1bceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:20.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.921+0000 7f1be05d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:20.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.921+0000 7f1be05d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:20.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.921+0000 7f1be05d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:20.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.921+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bd8069f40 con 0x7f1bd81430c0 2024-03-25T18:35:20.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.947+0000 7f1bceffd640 1 -- 172.21.15.55:0/2440890678 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f1bd0052670 con 0x7f1bd81430c0 2024-03-25T18:35:20.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.947+0000 7f1bceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:20.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.947+0000 7f1bceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:20.948 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:35:20.948 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 10 librados: watch_flush enter 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 10 librados: watch_flush exit 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 10 client.4557.objecter successfully canceled tick 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 >> v1:172.21.15.55:6800/286536064 conn(0x7f1bbc06dbf0 legacy=0x7f1bbc0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 10 monclient: shutdown 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 >> v1:172.21.15.55:6789/0 conn(0x7f1bd81430c0 legacy=0x7f1bd81e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 shutdown_connections 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.949+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 >> 172.21.15.55:0/2440890678 conn(0x7f1bd80b7ea0 msgr2=0x7f1bd8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.950+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 shutdown_connections 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.950+0000 7f1be05d6640 1 -- 172.21.15.55:0/2440890678 wait complete. 2024-03-25T18:35:20.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:20.950+0000 7f1be05d6640 1 librados: shutdown 2024-03-25T18:35:20.962 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:27.768 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:27.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.884+0000 7ff14d9d6640 10 monclient: get_monmap_and_config 2024-03-25T18:35:27.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.884+0000 7ff14d9d6640 10 monclient: build_initial_monmap 2024-03-25T18:35:27.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.884+0000 7ff14d9d6640 10 monclient: monmap: 2024-03-25T18:35:27.886 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:27.886 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:27.886 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:35:27.885078+0000 2024-03-25T18:35:27.886 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:35:27.885078+0000 2024-03-25T18:35:27.886 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:27.886 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:27.886 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:27.887 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:35:27.887 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:35:27.887 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:27.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.884+0000 7ff14d9d6640 1 Processor -- start 2024-03-25T18:35:27.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.884+0000 7ff14d9d6640 1 -- start start 2024-03-25T18:35:27.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.884+0000 7ff14d9d6640 10 monclient: init 2024-03-25T18:35:27.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:27.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:27.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff1480a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:27.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff14813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:27.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff1481430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:27.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff148142520 con 0x7ff1481430c0 2024-03-25T18:35:27.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1480b28a0 con 0x7ff14813f710 2024-03-25T18:35:27.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1480b2a30 con 0x7ff1480a98f0 2024-03-25T18:35:27.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:27.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.885+0000 7ff14d9d6640 10 monclient(hunting): authenticate will time out at 7732.143332s 2024-03-25T18:35:27.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.889+0000 7ff1477fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ff1481430c0 0x7ff14814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47900/0 (socket says 172.21.15.55:47900) 2024-03-25T18:35:27.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.889+0000 7ff1477fe640 1 -- 172.21.15.55:0/1844613647 learned_addr learned my addr 172.21.15.55:0/1844613647 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:27.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.893+0000 7ff145ffb640 1 -- 172.21.15.55:0/1844613647 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787168097 0 0) 0x7ff148142520 con 0x7ff1481430c0 2024-03-25T18:35:27.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.893+0000 7ff145ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:27.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.893+0000 7ff145ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:27.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.893+0000 7ff145ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:27.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.893+0000 7ff145ffb640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:35:27.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.893+0000 7ff145ffb640 1 -- 172.21.15.55:0/1844613647 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff130003520 con 0x7ff1481430c0 2024-03-25T18:35:27.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.900+0000 7ff145ffb640 1 -- 172.21.15.55:0/1844613647 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1770702523 0 0) 0x7ff130003520 con 0x7ff1481430c0 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 1 -- 172.21.15.55:0/1844613647 >> v1:172.21.15.55:6790/0 conn(0x7ff1480a98f0 legacy=0x7ff1480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 1 -- 172.21.15.55:0/1844613647 >> v1:172.21.15.133:6789/0 conn(0x7ff14813f710 legacy=0x7ff1481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 1 monclient: found mon.noname-a 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 1 -- 172.21.15.55:0/1844613647 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1480b44c0 con 0x7ff1481430c0 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 1 -- 172.21.15.55:0/1844613647 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff140001690 con 0x7ff1481430c0 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:27.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff145ffb640 10 monclient: dump: 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:27.903 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:27.904 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:27.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff14d9d6640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:35:27.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.901+0000 7ff14d9d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:27.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.905+0000 7ff145ffb640 1 -- 172.21.15.55:0/1844613647 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff140001860 con 0x7ff1481430c0 2024-03-25T18:35:27.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.905+0000 7ff145ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:27.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.905+0000 7ff145ffb640 1 -- 172.21.15.55:0/1844613647 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff140001cb0 con 0x7ff1481430c0 2024-03-25T18:35:27.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.905+0000 7ff14d9d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:27.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.905+0000 7ff14d9d6640 4 set_mon_vals no callback set 2024-03-25T18:35:27.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.905+0000 7ff145ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:27.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.905+0000 7ff145ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.905+0000 7ff145ffb640 10 monclient: dump: 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:27.907 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:27.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.906+0000 7ff14d9d6640 10 monclient: shutdown 2024-03-25T18:35:27.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.906+0000 7ff14d9d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:27.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.906+0000 7ff14d9d6640 1 -- 172.21.15.55:0/1844613647 >> v1:172.21.15.55:6789/0 conn(0x7ff1481430c0 legacy=0x7ff14814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:27.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.906+0000 7ff14d9d6640 1 -- 172.21.15.55:0/1844613647 shutdown_connections 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.906+0000 7ff14d9d6640 1 -- 172.21.15.55:0/1844613647 >> 172.21.15.55:0/1844613647 conn(0x7ff1480b7ea0 msgr2=0x7ff1480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.906+0000 7ff14d9d6640 1 -- 172.21.15.55:0/1844613647 shutdown_connections 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.906+0000 7ff14d9d6640 1 -- 172.21.15.55:0/1844613647 wait complete. 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.907+0000 7ff14d9d6640 10 monclient: build_initial_monmap 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.907+0000 7ff14d9d6640 10 monclient: monmap: 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:27.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.907+0000 7ff14d9d6640 1 librados: starting msgr at 2024-03-25T18:35:27.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.907+0000 7ff14d9d6640 1 librados: starting objecter 2024-03-25T18:35:27.910 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.908+0000 7ff14d9d6640 1 Processor -- start 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.908+0000 7ff14d9d6640 1 -- start start 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.908+0000 7ff14d9d6640 1 librados: setting wanted keys 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.908+0000 7ff14d9d6640 1 librados: calling monclient init 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.908+0000 7ff14d9d6640 10 monclient: init 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff1480a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff14813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff1481430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14813afc0 con 0x7ff1481430c0 2024-03-25T18:35:27.911 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14813b2a0 con 0x7ff14813f710 2024-03-25T18:35:27.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14813b580 con 0x7ff1480a98f0 2024-03-25T18:35:27.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:27.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.909+0000 7ff14d9d6640 10 monclient(hunting): authenticate will time out at 7732.166862s 2024-03-25T18:35:27.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.920+0000 7ff1477fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7ff1481430c0 0x7ff1481e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47908/0 (socket says 172.21.15.55:47908) 2024-03-25T18:35:27.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.920+0000 7ff1477fe640 1 -- 172.21.15.55:0/3244283155 learned_addr learned my addr 172.21.15.55:0/3244283155 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:27.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.922+0000 7ff12ffff640 10 client.?.objecter ms_handle_connect 0x7ff14813f710 2024-03-25T18:35:27.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.922+0000 7ff12ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:27.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.924+0000 7ff12ffff640 10 client.?.objecter ms_handle_connect 0x7ff1480a98f0 2024-03-25T18:35:27.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.924+0000 7ff12ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:27.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.926+0000 7ff12ffff640 10 client.?.objecter ms_handle_connect 0x7ff1481430c0 2024-03-25T18:35:27.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.926+0000 7ff12ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:27.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.932+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464514897 0 0) 0x7ff14813b580 con 0x7ff1480a98f0 2024-03-25T18:35:27.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.932+0000 7ff12ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:27.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.932+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:27.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.932+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:27.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.932+0000 7ff12ffff640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:35:27.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.932+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff134004eb0 con 0x7ff1480a98f0 2024-03-25T18:35:27.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.940+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122477159 0 0) 0x7ff14813afc0 con 0x7ff1481430c0 2024-03-25T18:35:27.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.940+0000 7ff12ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:27.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.940+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:27.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.940+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:27.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.940+0000 7ff12ffff640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:35:27.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.940+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14813b580 con 0x7ff1481430c0 2024-03-25T18:35:27.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.941+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326797477 0 0) 0x7ff14813b2a0 con 0x7ff14813f710 2024-03-25T18:35:27.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.941+0000 7ff12ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:27.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.941+0000 7ff12ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:27.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.941+0000 7ff12ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:27.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.941+0000 7ff12ffff640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:35:27.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.941+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14813afc0 con 0x7ff14813f710 2024-03-25T18:35:27.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.943+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1182940579 0 0) 0x7ff134004eb0 con 0x7ff1480a98f0 2024-03-25T18:35:27.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.943+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff14813b2a0 con 0x7ff1480a98f0 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.954+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff138001cb0 con 0x7ff1480a98f0 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.954+0000 7ff12ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.954+0000 7ff12ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.954+0000 7ff12ffff640 10 monclient(hunting): dump: 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:27.955 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:27.956 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:27.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.954+0000 7ff12ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:27.963 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.960+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1457789302 0 0) 0x7ff14813b2a0 con 0x7ff1480a98f0 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 >> v1:172.21.15.133:6789/0 conn(0x7ff14813f710 legacy=0x7ff1481df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 >> v1:172.21.15.55:6789/0 conn(0x7ff1481430c0 legacy=0x7ff1481e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 1 monclient: found mon.c 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1481390b0 con 0x7ff1480a98f0 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 10 monclient: _finish_auth 0 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff12ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff14d9d6640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff14d9d6640 10 monclient: _renew_subs 2024-03-25T18:35:27.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff14d9d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1480b7740 con 0x7ff1480a98f0 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff14d9d6640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff14d9d6640 10 monclient: _renew_subs 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff14d9d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1480b5a70 con 0x7ff1480a98f0 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.961+0000 7ff14d9d6640 1 librados: init done 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.962+0000 7ff14d9d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.962+0000 7ff14d9d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.962+0000 7ff14d9d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:27.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:27.962+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1480b7930 con 0x7ff1480a98f0 2024-03-25T18:35:28.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.284+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff138003970 con 0x7ff1480a98f0 2024-03-25T18:35:28.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.284+0000 7ff12ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.284+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7ff138003c80 con 0x7ff1480a98f0 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.284+0000 7ff12ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.284+0000 7ff12ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.284+0000 7ff12ffff640 10 monclient: dump: 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:28.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.284+0000 7ff1457fa640 4 set_mon_vals no callback set 2024-03-25T18:35:28.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.285+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7ff138019c50 con 0x7ff1480a98f0 2024-03-25T18:35:28.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.285+0000 7ff1467fc640 10 monclient: get_auth_request con 0x7ff13406ddb0 auth_method 0 2024-03-25T18:35:28.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.286+0000 7ff12ffff640 10 client.4547.objecter ms_handle_connect 0x7ff13406ddb0 2024-03-25T18:35:28.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.290+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7ff138053960 con 0x7ff1480a98f0 2024-03-25T18:35:28.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.290+0000 7ff12ffff640 10 client.4547.objecter ms_dispatch 0x7ff148000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:28.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.290+0000 7ff12ffff640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:28.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.290+0000 7ff12ffff640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:28.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.290+0000 7ff12ffff640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-25T18:35:28.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.290+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff13804f2a0 con 0x7ff1480a98f0 2024-03-25T18:35:28.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.290+0000 7ff12ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:28.292 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.290+0000 7ff12ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:28.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.566+0000 7ff14d9d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:28.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.566+0000 7ff14d9d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:28.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.566+0000 7ff14d9d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:28.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.566+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff14814f2d0 con 0x7ff1480a98f0 2024-03-25T18:35:28.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.587+0000 7ff12ffff640 1 -- 172.21.15.55:0/3244283155 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7ff1380513a0 con 0x7ff1480a98f0 2024-03-25T18:35:28.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.587+0000 7ff12ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:28.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.587+0000 7ff12ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:28.588 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:35:28.588 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 10 librados: watch_flush enter 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 10 librados: watch_flush exit 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 10 client.4547.objecter successfully canceled tick 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 >> v1:172.21.15.55:6800/286536064 conn(0x7ff13406ddb0 legacy=0x7ff134070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 10 monclient: shutdown 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 >> v1:172.21.15.55:6790/0 conn(0x7ff1480a98f0 legacy=0x7ff148139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 shutdown_connections 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 >> 172.21.15.55:0/3244283155 conn(0x7ff1480b7ea0 msgr2=0x7ff148074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 shutdown_connections 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 1 -- 172.21.15.55:0/3244283155 wait complete. 2024-03-25T18:35:28.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:28.589+0000 7ff14d9d6640 1 librados: shutdown 2024-03-25T18:35:28.601 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:35.408 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.522+0000 7f69b664e640 10 monclient: get_monmap_and_config 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.522+0000 7f69b664e640 10 monclient: build_initial_monmap 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.522+0000 7f69b664e640 10 monclient: monmap: 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:35:35.522933+0000 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:35:35.522933+0000 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:35:35.523 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:35:35.524 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.522+0000 7f69b664e640 1 Processor -- start 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 1 -- start start 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 10 monclient: init 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 10 monclient(hunting): picked mon.noname-a con 0x7f69b00a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 10 monclient(hunting): picked mon.noname-b con 0x7f69b013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 10 monclient(hunting): picked mon.noname-c con 0x7f69b01430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b0142520 con 0x7f69b00a98f0 2024-03-25T18:35:35.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b00b28a0 con 0x7f69b01430c0 2024-03-25T18:35:35.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b00b2a30 con 0x7f69b013f710 2024-03-25T18:35:35.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:35.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.523+0000 7f69b664e640 10 monclient(hunting): authenticate will time out at 7739.781104s 2024-03-25T18:35:35.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.544+0000 7f69b4beb640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f69b01430c0 0x7f69b014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:35524/0 (socket says 172.21.15.55:35524) 2024-03-25T18:35:35.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.544+0000 7f69b4beb640 1 -- 172.21.15.55:0/1585050580 learned_addr learned my addr 172.21.15.55:0/1585050580 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:35.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.563+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547444342 0 0) 0x7f69b00b28a0 con 0x7f69b01430c0 2024-03-25T18:35:35.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.563+0000 7f69aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:35.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.563+0000 7f69aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:35.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.563+0000 7f69aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:35.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.563+0000 7f69aeffd640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:35:35.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.563+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6998003520 con 0x7f69b01430c0 2024-03-25T18:35:35.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.574+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713378820 0 0) 0x7f69b0142520 con 0x7f69b00a98f0 2024-03-25T18:35:35.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.574+0000 7f69aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:35.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.574+0000 7f69aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:35.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.574+0000 7f69aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:35.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.574+0000 7f69aeffd640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:35:35.575 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.574+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b00b28a0 con 0x7f69b00a98f0 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3219088753 0 0) 0x7f6998003520 con 0x7f69b01430c0 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 >> v1:172.21.15.55:6790/0 conn(0x7f69b013f710 legacy=0x7f69b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 >> v1:172.21.15.55:6789/0 conn(0x7f69b00a98f0 legacy=0x7f69b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 1 monclient: found mon.noname-c 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b00b44c0 con 0x7f69b01430c0 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:35.579 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f69a8001a10 con 0x7f69b01430c0 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69aeffd640 10 monclient: dump: 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69b664e640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:35:35.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.577+0000 7f69b664e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.587+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69a8001b70 con 0x7f69b01430c0 2024-03-25T18:35:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.587+0000 7f69aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.587+0000 7f69aeffd640 1 -- 172.21.15.55:0/1585050580 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f69a8001f80 con 0x7f69b01430c0 2024-03-25T18:35:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.587+0000 7f69aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.587+0000 7f69aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:35.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.587+0000 7f69aeffd640 10 monclient: dump: 2024-03-25T18:35:35.588 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:35.588 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:35.588 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:35.589 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:35.589 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:35.589 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:35.589 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:35.589 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:35.589 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:35.589 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:35.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.587+0000 7f69b664e640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:35.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.587+0000 7f69b664e640 4 set_mon_vals no callback set 2024-03-25T18:35:35.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.588+0000 7f69b664e640 10 monclient: shutdown 2024-03-25T18:35:35.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.588+0000 7f69b664e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.588+0000 7f69b664e640 1 -- 172.21.15.55:0/1585050580 >> v1:172.21.15.133:6789/0 conn(0x7f69b01430c0 legacy=0x7f69b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.588+0000 7f69b664e640 1 -- 172.21.15.55:0/1585050580 shutdown_connections 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.588+0000 7f69b664e640 1 -- 172.21.15.55:0/1585050580 >> 172.21.15.55:0/1585050580 conn(0x7f69b00b7ea0 msgr2=0x7f69b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.588+0000 7f69b664e640 1 -- 172.21.15.55:0/1585050580 shutdown_connections 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.588+0000 7f69b664e640 1 -- 172.21.15.55:0/1585050580 wait complete. 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.588+0000 7f69b664e640 10 monclient: build_initial_monmap 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.588+0000 7f69b664e640 10 monclient: monmap: 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:35:35.592 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.589+0000 7f69b664e640 1 librados: starting msgr at 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.589+0000 7f69b664e640 1 librados: starting objecter 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.589+0000 7f69b664e640 1 Processor -- start 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.589+0000 7f69b664e640 1 -- start start 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.589+0000 7f69b664e640 1 librados: setting wanted keys 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.589+0000 7f69b664e640 1 librados: calling monclient init 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.589+0000 7f69b664e640 10 monclient: init 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 10 monclient(hunting): picked mon.noname-c con 0x7f69b00a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 10 monclient(hunting): picked mon.noname-b con 0x7f69b013f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 10 monclient(hunting): picked mon.noname-a con 0x7f69b01430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:35.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b0138100 con 0x7f69b01430c0 2024-03-25T18:35:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b0139890 con 0x7f69b013f710 2024-03-25T18:35:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b0138a10 con 0x7f69b00a98f0 2024-03-25T18:35:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:35.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.590+0000 7f69b664e640 10 monclient(hunting): authenticate will time out at 7739.847882s 2024-03-25T18:35:35.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.595+0000 7f69affff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f69b00a98f0 0x7f69b01379f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:39172/0 (socket says 172.21.15.55:39172) 2024-03-25T18:35:35.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.595+0000 7f69affff640 1 -- 172.21.15.55:0/4143587586 learned_addr learned my addr 172.21.15.55:0/4143587586 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:35.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.597+0000 7f69acff9640 10 client.?.objecter ms_handle_connect 0x7f69b00a98f0 2024-03-25T18:35:35.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.597+0000 7f69acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:35.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.603+0000 7f69acff9640 10 client.?.objecter ms_handle_connect 0x7f69b013f710 2024-03-25T18:35:35.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.603+0000 7f69acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:35.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.604+0000 7f69acff9640 10 client.?.objecter ms_handle_connect 0x7f69b01430c0 2024-03-25T18:35:35.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.604+0000 7f69acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:35.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.605+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244849766 0 0) 0x7f69b0138a10 con 0x7f69b00a98f0 2024-03-25T18:35:35.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.606+0000 7f69acff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:35.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.606+0000 7f69acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:35.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.606+0000 7f69acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:35.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.606+0000 7f69acff9640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:35:35.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.606+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f699c004eb0 con 0x7f69b00a98f0 2024-03-25T18:35:35.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.612+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378755925 0 0) 0x7f69b0138100 con 0x7f69b01430c0 2024-03-25T18:35:35.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.612+0000 7f69acff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:35.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.612+0000 7f69acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:35.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.612+0000 7f69acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:35.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.612+0000 7f69acff9640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:35:35.613 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.612+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b0138a10 con 0x7f69b01430c0 2024-03-25T18:35:35.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.615+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 470916552 0 0) 0x7f699c004eb0 con 0x7f69b00a98f0 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.615+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69b0138100 con 0x7f69b00a98f0 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.615+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f69a0001a10 con 0x7f69b00a98f0 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.615+0000 7f69acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.615+0000 7f69acff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.615+0000 7f69acff9640 10 monclient(hunting): dump: 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:35.617 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.615+0000 7f69acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:35.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 618173782 0 0) 0x7f69b0138100 con 0x7f69b00a98f0 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 >> v1:172.21.15.133:6789/0 conn(0x7f69b013f710 legacy=0x7f69b01d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 >> v1:172.21.15.55:6789/0 conn(0x7f69b01430c0 legacy=0x7f69b01d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 10 monclient: _finish_hunting 0 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 1 monclient: found mon.c 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b01382b0 con 0x7f69b00a98f0 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 10 monclient: _finish_auth 0 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 10 monclient: _check_auth_tickets 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69b664e640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69b664e640 10 monclient: _renew_subs 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69b664e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:35.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b0138560 con 0x7f69b00a98f0 2024-03-25T18:35:35.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69b664e640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:35.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69b664e640 10 monclient: _renew_subs 2024-03-25T18:35:35.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69b664e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:35.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b01e31c0 con 0x7f69b00a98f0 2024-03-25T18:35:35.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.623+0000 7f69b664e640 1 librados: init done 2024-03-25T18:35:35.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.624+0000 7f69b664e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:35.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.624+0000 7f69b664e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:35.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.624+0000 7f69b664e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:35.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.624+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b0138ba0 con 0x7f69b00a98f0 2024-03-25T18:35:35.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.938+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69a0001c50 con 0x7f69b00a98f0 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.938+0000 7f69acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.938+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f69a0003a60 con 0x7f69b00a98f0 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.938+0000 7f69acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.939+0000 7f69acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.939+0000 7f69acff9640 10 monclient: dump: 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:35.940 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:35.941 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:35.941 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:35.941 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:35.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.939+0000 7f69ae7fc640 4 set_mon_vals no callback set 2024-03-25T18:35:35.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.939+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f69a0003c80 con 0x7f69b00a98f0 2024-03-25T18:35:35.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.940+0000 7f69af7fe640 10 monclient: get_auth_request con 0x7f699c06dd10 auth_method 0 2024-03-25T18:35:35.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.940+0000 7f69acff9640 10 client.4553.objecter ms_handle_connect 0x7f699c06dd10 2024-03-25T18:35:35.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.946+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f69a0054860 con 0x7f69b00a98f0 2024-03-25T18:35:35.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.946+0000 7f69acff9640 10 client.4553.objecter ms_dispatch 0x7f69b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:35.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.946+0000 7f69acff9640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:35.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.946+0000 7f69acff9640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:35.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.946+0000 7f69acff9640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-25T18:35:35.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.946+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69a0050370 con 0x7f69b00a98f0 2024-03-25T18:35:35.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.947+0000 7f69acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:35.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:35.947+0000 7f69acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:36.152 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:07:46.322 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:36.160 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:07:46.330 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.164+0000 7f69b664e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.164+0000 7f69b664e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.164+0000 7f69b664e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:36.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.164+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69b00627b0 con 0x7f69b00a98f0 2024-03-25T18:35:36.199 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:07:46.369 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:36.200 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:46.338 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:36.208 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:46.345 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:36.216 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:07:46.386 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:36.238 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:07:46.408 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:36.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.244+0000 7f69acff9640 1 -- 172.21.15.55:0/4143587586 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f69a00522a0 con 0x7f69b00a98f0 2024-03-25T18:35:36.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.244+0000 7f69acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:36.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.244+0000 7f69acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:36.245 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:35:36.245 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:36.246 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:07:46.383 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 10 librados: watch_flush enter 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 10 librados: watch_flush exit 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 10 client.4553.objecter successfully canceled tick 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 >> v1:172.21.15.55:6800/286536064 conn(0x7f699c06dd10 legacy=0x7f699c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 10 monclient: shutdown 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 >> v1:172.21.15.55:6790/0 conn(0x7f69b00a98f0 legacy=0x7f69b01379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 shutdown_connections 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.246+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 >> 172.21.15.55:0/4143587586 conn(0x7f69b00b7ea0 msgr2=0x7f69b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:36.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.247+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 shutdown_connections 2024-03-25T18:35:36.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.247+0000 7f69b664e640 1 -- 172.21.15.55:0/4143587586 wait complete. 2024-03-25T18:35:36.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:36.247+0000 7f69b664e640 1 librados: shutdown 2024-03-25T18:35:36.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:36.269 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:07:46.439 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:43.067 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.187+0000 7fb575dcd640 10 monclient: get_monmap_and_config 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.187+0000 7fb575dcd640 10 monclient: build_initial_monmap 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.187+0000 7fb575dcd640 10 monclient: monmap: 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:35:43.187932+0000 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:35:43.187932+0000 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:43.188 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:43.189 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:35:43.189 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:35:43.189 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:43.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.188+0000 7fb575dcd640 1 Processor -- start 2024-03-25T18:35:43.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.188+0000 7fb575dcd640 1 -- start start 2024-03-25T18:35:43.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.188+0000 7fb575dcd640 10 monclient: init 2024-03-25T18:35:43.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:43.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:43.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 10 monclient(hunting): picked mon.noname-c con 0x7fb570070410 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:43.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 10 monclient(hunting): picked mon.noname-a con 0x7fb5700707f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:43.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 10 monclient(hunting): picked mon.noname-b con 0x7fb57013db40 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:43.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5700adea0 con 0x7fb5700707f0 2024-03-25T18:35:43.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5701445a0 con 0x7fb570070410 2024-03-25T18:35:43.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5700720a0 con 0x7fb57013db40 2024-03-25T18:35:43.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:43.191 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.189+0000 7fb575dcd640 10 monclient(hunting): authenticate will time out at 7747.446786s 2024-03-25T18:35:43.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.196+0000 7fb56ffff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fb57013db40 0x7fb57014ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:57484/0 (socket says 172.21.15.55:57484) 2024-03-25T18:35:43.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.196+0000 7fb56ffff640 1 -- 172.21.15.55:0/700072377 learned_addr learned my addr 172.21.15.55:0/700072377 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:43.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.203+0000 7fb56e7fc640 1 -- 172.21.15.55:0/700072377 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64575140 0 0) 0x7fb5700adea0 con 0x7fb5700707f0 2024-03-25T18:35:43.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.203+0000 7fb56e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:43.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.203+0000 7fb56e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:43.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.203+0000 7fb56e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:43.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.203+0000 7fb56e7fc640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:35:43.204 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.203+0000 7fb56e7fc640 1 -- 172.21.15.55:0/700072377 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb558003520 con 0x7fb5700707f0 2024-03-25T18:35:43.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 1 -- 172.21.15.55:0/700072377 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971480371 0 0) 0x7fb558003520 con 0x7fb5700707f0 2024-03-25T18:35:43.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 1 -- 172.21.15.55:0/700072377 >> v1:172.21.15.55:6790/0 conn(0x7fb57013db40 legacy=0x7fb57014ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 1 -- 172.21.15.55:0/700072377 >> v1:172.21.15.133:6789/0 conn(0x7fb570070410 legacy=0x7fb570143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:43.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:35:43.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:43.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:35:43.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 1 -- 172.21.15.55:0/700072377 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb570072380 con 0x7fb5700707f0 2024-03-25T18:35:43.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 1 -- 172.21.15.55:0/700072377 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb564001690 con 0x7fb5700707f0 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb56e7fc640 10 monclient: dump: 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:43.212 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:43.213 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:43.213 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:43.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb575dcd640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:35:43.213 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.210+0000 7fb575dcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:43.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.216+0000 7fb56e7fc640 1 -- 172.21.15.55:0/700072377 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5640017f0 con 0x7fb5700707f0 2024-03-25T18:35:43.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.216+0000 7fb56e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:43.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.216+0000 7fb56e7fc640 1 -- 172.21.15.55:0/700072377 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb564001c00 con 0x7fb5700707f0 2024-03-25T18:35:43.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.216+0000 7fb56e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:43.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.216+0000 7fb56e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:43.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.216+0000 7fb56e7fc640 10 monclient: dump: 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.216+0000 7fb575dcd640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:43.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.216+0000 7fb575dcd640 4 set_mon_vals no callback set 2024-03-25T18:35:43.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.217+0000 7fb575dcd640 10 monclient: shutdown 2024-03-25T18:35:43.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.217+0000 7fb575dcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:43.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.217+0000 7fb575dcd640 1 -- 172.21.15.55:0/700072377 >> v1:172.21.15.55:6789/0 conn(0x7fb5700707f0 legacy=0x7fb57013d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.217+0000 7fb575dcd640 1 -- 172.21.15.55:0/700072377 shutdown_connections 2024-03-25T18:35:43.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.217+0000 7fb575dcd640 1 -- 172.21.15.55:0/700072377 >> 172.21.15.55:0/700072377 conn(0x7fb57006b630 msgr2=0x7fb5700b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:43.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.217+0000 7fb575dcd640 1 -- 172.21.15.55:0/700072377 shutdown_connections 2024-03-25T18:35:43.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.217+0000 7fb575dcd640 1 -- 172.21.15.55:0/700072377 wait complete. 2024-03-25T18:35:43.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.218+0000 7fb575dcd640 10 monclient: build_initial_monmap 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.218+0000 7fb575dcd640 10 monclient: monmap: 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.218+0000 7fb575dcd640 1 librados: starting msgr at 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.218+0000 7fb575dcd640 1 librados: starting objecter 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.219+0000 7fb575dcd640 1 Processor -- start 2024-03-25T18:35:43.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.219+0000 7fb575dcd640 1 -- start start 2024-03-25T18:35:43.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.219+0000 7fb575dcd640 1 librados: setting wanted keys 2024-03-25T18:35:43.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.219+0000 7fb575dcd640 1 librados: calling monclient init 2024-03-25T18:35:43.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.219+0000 7fb575dcd640 10 monclient: init 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.220+0000 7fb575dcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.220+0000 7fb575dcd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.220+0000 7fb575dcd640 10 monclient(hunting): picked mon.noname-a con 0x7fb570070410 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.220+0000 7fb575dcd640 10 monclient(hunting): picked mon.noname-c con 0x7fb5700707f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.220+0000 7fb575dcd640 10 monclient(hunting): picked mon.noname-b con 0x7fb57013db40 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.220+0000 7fb575dcd640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57006dca0 con 0x7fb570070410 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.220+0000 7fb575dcd640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57006df80 con 0x7fb57013db40 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.221+0000 7fb575dcd640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb570143660 con 0x7fb5700707f0 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.221+0000 7fb575dcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:43.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.221+0000 7fb575dcd640 10 monclient(hunting): authenticate will time out at 7747.478389s 2024-03-25T18:35:43.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.226+0000 7fb56f7fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fb570070410 0x7fb570135c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:41360/0 (socket says 172.21.15.55:41360) 2024-03-25T18:35:43.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.227+0000 7fb56f7fe640 1 -- 172.21.15.55:0/964855469 learned_addr learned my addr 172.21.15.55:0/964855469 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:43.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.228+0000 7fb557fff640 10 client.?.objecter ms_handle_connect 0x7fb570070410 2024-03-25T18:35:43.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.228+0000 7fb557fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:43.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.234+0000 7fb557fff640 10 client.?.objecter ms_handle_connect 0x7fb5700707f0 2024-03-25T18:35:43.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.234+0000 7fb557fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:43.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.235+0000 7fb557fff640 10 client.?.objecter ms_handle_connect 0x7fb57013db40 2024-03-25T18:35:43.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.235+0000 7fb557fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:43.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.242+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650876970 0 0) 0x7fb570143660 con 0x7fb5700707f0 2024-03-25T18:35:43.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.243+0000 7fb557fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:43.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.243+0000 7fb557fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:43.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.243+0000 7fb557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:43.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.243+0000 7fb557fff640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:35:43.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.243+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb55c004eb0 con 0x7fb5700707f0 2024-03-25T18:35:43.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.245+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290978977 0 0) 0x7fb57006df80 con 0x7fb57013db40 2024-03-25T18:35:43.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.245+0000 7fb557fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:43.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.245+0000 7fb557fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:43.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.245+0000 7fb557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:43.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.245+0000 7fb557fff640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:35:43.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.245+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb570143660 con 0x7fb57013db40 2024-03-25T18:35:43.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.255+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884065778 0 0) 0x7fb57006dca0 con 0x7fb570070410 2024-03-25T18:35:43.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.255+0000 7fb557fff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:43.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.255+0000 7fb557fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:43.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.255+0000 7fb557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:43.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.255+0000 7fb557fff640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:35:43.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.255+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb57006df80 con 0x7fb570070410 2024-03-25T18:35:43.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.259+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1742831461 0 0) 0x7fb570143660 con 0x7fb57013db40 2024-03-25T18:35:43.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.259+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb57006dca0 con 0x7fb57013db40 2024-03-25T18:35:43.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.259+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb568001b00 con 0x7fb57013db40 2024-03-25T18:35:43.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.259+0000 7fb557fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:43.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.259+0000 7fb557fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:43.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.259+0000 7fb557fff640 10 monclient(hunting): dump: 2024-03-25T18:35:43.260 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:43.260 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:43.261 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.261 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.261 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:43.261 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:43.261 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:43.261 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:43.261 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:43.261 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:43.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.259+0000 7fb557fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:43.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.261+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2498099291 0 0) 0x7fb55c004eb0 con 0x7fb5700707f0 2024-03-25T18:35:43.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.261+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb570143660 con 0x7fb5700707f0 2024-03-25T18:35:43.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.261+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb564001bc0 con 0x7fb5700707f0 2024-03-25T18:35:43.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.261+0000 7fb557fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:43.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.261+0000 7fb557fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:43.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.261+0000 7fb557fff640 10 monclient(hunting): dump: 2024-03-25T18:35:43.262 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:43.263 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:43.263 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.263 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.263 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:43.263 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:43.263 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:43.263 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:43.263 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:43.263 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 481348894 0 0) 0x7fb570143660 con 0x7fb5700707f0 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 >> v1:172.21.15.133:6789/0 conn(0x7fb57013db40 legacy=0x7fb5701db0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 >> v1:172.21.15.55:6789/0 conn(0x7fb570070410 legacy=0x7fb570135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 1 monclient: found mon.c 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57006e410 con 0x7fb5700707f0 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 10 monclient: _finish_auth 0 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.273+0000 7fb557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:43.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.274+0000 7fb575dcd640 5 monclient: authenticate success, global_id 4559 2024-03-25T18:35:43.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.274+0000 7fb575dcd640 10 monclient: _renew_subs 2024-03-25T18:35:43.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.274+0000 7fb575dcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:43.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.274+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57006e840 con 0x7fb5700707f0 2024-03-25T18:35:43.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.274+0000 7fb575dcd640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:43.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.274+0000 7fb575dcd640 10 monclient: _renew_subs 2024-03-25T18:35:43.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.274+0000 7fb575dcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:43.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.274+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb570142950 con 0x7fb5700707f0 2024-03-25T18:35:43.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.274+0000 7fb575dcd640 1 librados: init done 2024-03-25T18:35:43.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.275+0000 7fb575dcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:43.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.275+0000 7fb575dcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:43.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.275+0000 7fb575dcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:43.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.275+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57006e660 con 0x7fb5700707f0 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.281+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5640038a0 con 0x7fb5700707f0 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.281+0000 7fb557fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.281+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fb564003bb0 con 0x7fb5700707f0 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.281+0000 7fb557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.281+0000 7fb557fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.281+0000 7fb557fff640 10 monclient: dump: 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.282 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:43.283 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:43.283 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:43.283 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:43.283 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:43.283 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:43.283 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:43.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.281+0000 7fb56dffb640 4 set_mon_vals no callback set 2024-03-25T18:35:43.580 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.580+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fb564019b80 con 0x7fb5700707f0 2024-03-25T18:35:43.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.580+0000 7fb56f7fe640 10 monclient: get_auth_request con 0x7fb55c06dfb0 auth_method 0 2024-03-25T18:35:43.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.580+0000 7fb557fff640 10 client.4559.objecter ms_handle_connect 0x7fb55c06dfb0 2024-03-25T18:35:43.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.585+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fb564053690 con 0x7fb5700707f0 2024-03-25T18:35:43.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.585+0000 7fb557fff640 10 client.4559.objecter ms_dispatch 0x7fb570000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:43.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.585+0000 7fb557fff640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:43.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.585+0000 7fb557fff640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:43.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.585+0000 7fb557fff640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-25T18:35:43.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.585+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb56404f1a0 con 0x7fb5700707f0 2024-03-25T18:35:43.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.586+0000 7fb557fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:43.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.586+0000 7fb557fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:43.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.810+0000 7fb575dcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:43.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.810+0000 7fb575dcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:43.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.810+0000 7fb575dcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:35:43.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.810+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5700627b0 con 0x7fb5700707f0 2024-03-25T18:35:43.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.874+0000 7fb557fff640 1 -- 172.21.15.55:0/964855469 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fb5640510d0 con 0x7fb5700707f0 2024-03-25T18:35:43.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.874+0000 7fb557fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:43.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.874+0000 7fb557fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:43.876 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:35:43.876 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:43.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 10 librados: watch_flush enter 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 10 librados: watch_flush exit 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 10 client.4559.objecter successfully canceled tick 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 >> v1:172.21.15.55:6800/286536064 conn(0x7fb55c06dfb0 legacy=0x7fb55c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 10 monclient: shutdown 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 >> v1:172.21.15.55:6790/0 conn(0x7fb5700707f0 legacy=0x7fb5701da9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 shutdown_connections 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.877+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 >> 172.21.15.55:0/964855469 conn(0x7fb57006b630 msgr2=0x7fb570070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.878+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 shutdown_connections 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.878+0000 7fb575dcd640 1 -- 172.21.15.55:0/964855469 wait complete. 2024-03-25T18:35:43.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:43.878+0000 7fb575dcd640 1 librados: shutdown 2024-03-25T18:35:43.891 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:50.698 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:50.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.812+0000 7fd0ce90e640 10 monclient: get_monmap_and_config 2024-03-25T18:35:50.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.812+0000 7fd0ce90e640 10 monclient: build_initial_monmap 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.812+0000 7fd0ce90e640 10 monclient: monmap: 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:35:50.813701+0000 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:35:50.813701+0000 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:35:50.814 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:50.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.813+0000 7fd0ce90e640 1 Processor -- start 2024-03-25T18:35:50.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 1 -- start start 2024-03-25T18:35:50.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 10 monclient: init 2024-03-25T18:35:50.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:50.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:50.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 10 monclient(hunting): picked mon.noname-a con 0x7fd0c80a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:50.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 10 monclient(hunting): picked mon.noname-b con 0x7fd0c813f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:50.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 10 monclient(hunting): picked mon.noname-c con 0x7fd0c81430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:50.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c8142520 con 0x7fd0c80a98f0 2024-03-25T18:35:50.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c80b28a0 con 0x7fd0c81430c0 2024-03-25T18:35:50.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c80b2a30 con 0x7fd0c813f710 2024-03-25T18:35:50.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:50.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.814+0000 7fd0ce90e640 10 monclient(hunting): authenticate will time out at 7755.072103s 2024-03-25T18:35:50.824 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.823+0000 7fd0c77fe640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fd0c813f710 0x7fd0c81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:49482/0 (socket says 172.21.15.55:49482) 2024-03-25T18:35:50.824 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.824+0000 7fd0c77fe640 1 -- 172.21.15.55:0/3711738892 learned_addr learned my addr 172.21.15.55:0/3711738892 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:50.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.834+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923908795 0 0) 0x7fd0c80b2a30 con 0x7fd0c813f710 2024-03-25T18:35:50.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.834+0000 7fd0c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:50.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.834+0000 7fd0c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:50.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.834+0000 7fd0c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:50.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.834+0000 7fd0c6ffd640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:35:50.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.834+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b0003520 con 0x7fd0c813f710 2024-03-25T18:35:50.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.842+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750537192 0 0) 0x7fd0c8142520 con 0x7fd0c80a98f0 2024-03-25T18:35:50.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.842+0000 7fd0c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:50.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.842+0000 7fd0c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:50.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.842+0000 7fd0c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:50.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.842+0000 7fd0c6ffd640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:35:50.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.842+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b0003840 con 0x7fd0c80a98f0 2024-03-25T18:35:50.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.852+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988786825 0 0) 0x7fd0c80b28a0 con 0x7fd0c81430c0 2024-03-25T18:35:50.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.852+0000 7fd0c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:50.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.852+0000 7fd0c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:50.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.852+0000 7fd0c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:50.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.852+0000 7fd0c6ffd640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:35:50.853 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.852+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c8142520 con 0x7fd0c81430c0 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.853+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 173293801 0 0) 0x7fd0b0003840 con 0x7fd0c80a98f0 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.853+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 >> v1:172.21.15.55:6790/0 conn(0x7fd0c813f710 legacy=0x7fd0c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.853+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 >> v1:172.21.15.133:6789/0 conn(0x7fd0c81430c0 legacy=0x7fd0c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c80b44c0 con 0x7fd0c80a98f0 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:50.855 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd0b8001670 con 0x7fd0c80a98f0 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0c6ffd640 10 monclient: dump: 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0ce90e640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:35:50.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.854+0000 7fd0ce90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:50.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.875+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0b80017d0 con 0x7fd0c80a98f0 2024-03-25T18:35:50.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.875+0000 7fd0c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:50.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.875+0000 7fd0ce90e640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.875+0000 7fd0ce90e640 4 set_mon_vals no callback set 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.875+0000 7fd0c6ffd640 1 -- 172.21.15.55:0/3711738892 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd0b8001be0 con 0x7fd0c80a98f0 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0c6ffd640 10 monclient: dump: 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:50.877 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0ce90e640 10 monclient: shutdown 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0ce90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0ce90e640 1 -- 172.21.15.55:0/3711738892 >> v1:172.21.15.55:6789/0 conn(0x7fd0c80a98f0 legacy=0x7fd0c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0ce90e640 1 -- 172.21.15.55:0/3711738892 shutdown_connections 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0ce90e640 1 -- 172.21.15.55:0/3711738892 >> 172.21.15.55:0/3711738892 conn(0x7fd0c80b7ea0 msgr2=0x7fd0c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0ce90e640 1 -- 172.21.15.55:0/3711738892 shutdown_connections 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.876+0000 7fd0ce90e640 1 -- 172.21.15.55:0/3711738892 wait complete. 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.877+0000 7fd0ce90e640 10 monclient: build_initial_monmap 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.877+0000 7fd0ce90e640 10 monclient: monmap: 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:50.879 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.877+0000 7fd0ce90e640 1 librados: starting msgr at 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.877+0000 7fd0ce90e640 1 librados: starting objecter 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.878+0000 7fd0ce90e640 1 Processor -- start 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.878+0000 7fd0ce90e640 1 -- start start 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.878+0000 7fd0ce90e640 1 librados: setting wanted keys 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.878+0000 7fd0ce90e640 1 librados: calling monclient init 2024-03-25T18:35:50.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.878+0000 7fd0ce90e640 10 monclient: init 2024-03-25T18:35:50.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:50.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:50.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 10 monclient(hunting): picked mon.noname-c con 0x7fd0c80a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:50.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 10 monclient(hunting): picked mon.noname-b con 0x7fd0c813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:50.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 10 monclient(hunting): picked mon.noname-a con 0x7fd0c81430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:50.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c81d7870 con 0x7fd0c81430c0 2024-03-25T18:35:50.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c81d7490 con 0x7fd0c813f710 2024-03-25T18:35:50.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c81d7620 con 0x7fd0c80a98f0 2024-03-25T18:35:50.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:50.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.879+0000 7fd0ce90e640 10 monclient(hunting): authenticate will time out at 7755.137135s 2024-03-25T18:35:50.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.885+0000 7fd0c77fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fd0c813f710 0x7fd0c81d7d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:46596/0 (socket says 172.21.15.55:46596) 2024-03-25T18:35:50.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.885+0000 7fd0c77fe640 1 -- 172.21.15.55:0/175743835 learned_addr learned my addr 172.21.15.55:0/175743835 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:50.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.885+0000 7fd0c4ff9640 10 client.?.objecter ms_handle_connect 0x7fd0c813f710 2024-03-25T18:35:50.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.885+0000 7fd0c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:50.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.886+0000 7fd0c4ff9640 10 client.?.objecter ms_handle_connect 0x7fd0c80a98f0 2024-03-25T18:35:50.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.886+0000 7fd0c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:50.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.888+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210746889 0 0) 0x7fd0c81d7490 con 0x7fd0c813f710 2024-03-25T18:35:50.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.888+0000 7fd0c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:50.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.888+0000 7fd0c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:50.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.888+0000 7fd0c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:50.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.888+0000 7fd0c4ff9640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:35:50.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.888+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b4004eb0 con 0x7fd0c813f710 2024-03-25T18:35:50.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.892+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245473082 0 0) 0x7fd0c81d7620 con 0x7fd0c80a98f0 2024-03-25T18:35:50.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.892+0000 7fd0c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:50.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.892+0000 7fd0c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:50.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.892+0000 7fd0c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:50.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.892+0000 7fd0c4ff9640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:35:50.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.892+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c81d7490 con 0x7fd0c80a98f0 2024-03-25T18:35:50.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.895+0000 7fd0c4ff9640 10 client.?.objecter ms_handle_connect 0x7fd0c81430c0 2024-03-25T18:35:50.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.895+0000 7fd0c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:50.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.895+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2711909033 0 0) 0x7fd0b4004eb0 con 0x7fd0c813f710 2024-03-25T18:35:50.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.895+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0c81d7620 con 0x7fd0c813f710 2024-03-25T18:35:50.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.895+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd0bc001a30 con 0x7fd0c813f710 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.895+0000 7fd0c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.895+0000 7fd0c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.895+0000 7fd0c4ff9640 10 monclient(hunting): dump: 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:50.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.895+0000 7fd0c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:50.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.915+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982266206 0 0) 0x7fd0c81d7870 con 0x7fd0c81430c0 2024-03-25T18:35:50.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.915+0000 7fd0c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:50.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.915+0000 7fd0c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:50.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.915+0000 7fd0c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:50.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.915+0000 7fd0c4ff9640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:35:50.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.915+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0b4004eb0 con 0x7fd0c81430c0 2024-03-25T18:35:50.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2141102685 0 0) 0x7fd0c81d7620 con 0x7fd0c813f710 2024-03-25T18:35:50.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 >> v1:172.21.15.55:6790/0 conn(0x7fd0c80a98f0 legacy=0x7fd0c8134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 >> v1:172.21.15.55:6789/0 conn(0x7fd0c81430c0 legacy=0x7fd0c8137940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 1 monclient: found mon.b 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c8138200 con 0x7fd0c813f710 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0ce90e640 5 monclient: authenticate success, global_id 4552 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0ce90e640 10 monclient: _renew_subs 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0ce90e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c81384b0 con 0x7fd0c813f710 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0ce90e640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0ce90e640 10 monclient: _renew_subs 2024-03-25T18:35:50.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0ce90e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:50.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c8134160 con 0x7fd0c813f710 2024-03-25T18:35:50.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.919+0000 7fd0ce90e640 1 librados: init done 2024-03-25T18:35:50.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.920+0000 7fd0ce90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:50.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.920+0000 7fd0ce90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:50.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.920+0000 7fd0ce90e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:50.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:50.920+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c81d7a00 con 0x7fd0c813f710 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.183+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0bc001b90 con 0x7fd0c813f710 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.183+0000 7fd0c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.183+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd0bc004c30 con 0x7fd0c813f710 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.183+0000 7fd0c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.184+0000 7fd0c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.184+0000 7fd0c4ff9640 10 monclient: dump: 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:51.185 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:51.186 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:51.186 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:51.186 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:51.186 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:51.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.184+0000 7fd0c67fc640 4 set_mon_vals no callback set 2024-03-25T18:35:51.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.184+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fd0bc01ac00 con 0x7fd0c813f710 2024-03-25T18:35:51.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.185+0000 7fd0c7fff640 10 monclient: get_auth_request con 0x7fd0b406de00 auth_method 0 2024-03-25T18:35:51.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.185+0000 7fd0c4ff9640 10 client.4552.objecter ms_handle_connect 0x7fd0b406de00 2024-03-25T18:35:51.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.189+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fd0bc054830 con 0x7fd0c813f710 2024-03-25T18:35:51.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.189+0000 7fd0c4ff9640 10 client.4552.objecter ms_dispatch 0x7fd0c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:51.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.189+0000 7fd0c4ff9640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:51.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.189+0000 7fd0c4ff9640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:51.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.189+0000 7fd0c4ff9640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T18:35:51.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.189+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd0bc050340 con 0x7fd0c813f710 2024-03-25T18:35:51.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.189+0000 7fd0c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:51.190 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.189+0000 7fd0c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:51.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.424+0000 7fd0ce90e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:51.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.424+0000 7fd0ce90e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:51.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.424+0000 7fd0ce90e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:35:51.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.424+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0c80627b0 con 0x7fd0c813f710 2024-03-25T18:35:51.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.511+0000 7fd0c4ff9640 1 -- 172.21.15.55:0/175743835 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fd0bc052270 con 0x7fd0c813f710 2024-03-25T18:35:51.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.511+0000 7fd0c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:51.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.511+0000 7fd0c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:51.513 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:35:51.513 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:51.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.513+0000 7fd0ce90e640 10 librados: watch_flush enter 2024-03-25T18:35:51.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.513+0000 7fd0ce90e640 10 librados: watch_flush exit 2024-03-25T18:35:51.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.513+0000 7fd0ce90e640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T18:35:51.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.513+0000 7fd0ce90e640 10 client.4552.objecter successfully canceled tick 2024-03-25T18:35:51.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.513+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 >> v1:172.21.15.55:6800/286536064 conn(0x7fd0b406de00 legacy=0x7fd0b40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:51.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.513+0000 7fd0ce90e640 10 monclient: shutdown 2024-03-25T18:35:51.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.513+0000 7fd0ce90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:51.515 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.513+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 >> v1:172.21.15.133:6789/0 conn(0x7fd0c813f710 legacy=0x7fd0c81d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:51.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.514+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 shutdown_connections 2024-03-25T18:35:51.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.514+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 >> 172.21.15.55:0/175743835 conn(0x7fd0c80b7ea0 msgr2=0x7fd0c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:51.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.514+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 shutdown_connections 2024-03-25T18:35:51.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.514+0000 7fd0ce90e640 1 -- 172.21.15.55:0/175743835 wait complete. 2024-03-25T18:35:51.516 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:51.514+0000 7fd0ce90e640 1 librados: shutdown 2024-03-25T18:35:51.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:56.206 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:08:06.375 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:56.263 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:08:06.433 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:56.295 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:08:06.465 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:56.306 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:06.443 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:56.318 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:06.456 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:56.321 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:08:06.491 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:56.335 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:08:06.505 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:56.361 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:08:06.530 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:56.362 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:06.499 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:58.333 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:58.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.433+0000 7f34203c6640 10 monclient: get_monmap_and_config 2024-03-25T18:35:58.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.433+0000 7f34203c6640 10 monclient: build_initial_monmap 2024-03-25T18:35:58.434 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.433+0000 7f34203c6640 10 monclient: monmap: 2024-03-25T18:35:58.434 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:58.434 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:58.434 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:35:58.434410+0000 2024-03-25T18:35:58.434 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:35:58.434410+0000 2024-03-25T18:35:58.434 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:58.435 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:58.435 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:58.435 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:35:58.435 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:35:58.435 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 1 Processor -- start 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 1 -- start start 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 10 monclient: init 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 10 monclient(hunting): picked mon.noname-b con 0x7f34180a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 10 monclient(hunting): picked mon.noname-a con 0x7f341813f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 10 monclient(hunting): picked mon.noname-c con 0x7f34181430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3418142520 con 0x7f341813f710 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180b28a0 con 0x7f34181430c0 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180b2a30 con 0x7f34180a98f0 2024-03-25T18:35:58.436 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.434+0000 7f34203c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:58.437 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.435+0000 7f34203c6640 10 monclient(hunting): authenticate will time out at 7762.692355s 2024-03-25T18:35:58.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.441+0000 7f341e963640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f34181430c0 0x7f341814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:51132/0 (socket says 172.21.15.55:51132) 2024-03-25T18:35:58.442 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.441+0000 7f341e963640 1 -- 172.21.15.55:0/440219791 learned_addr learned my addr 172.21.15.55:0/440219791 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:58.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124994754 0 0) 0x7f3418142520 con 0x7f341813f710 2024-03-25T18:35:58.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:35:58.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404003520 con 0x7f341813f710 2024-03-25T18:35:58.454 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477945573 0 0) 0x7f34180b2a30 con 0x7f34180a98f0 2024-03-25T18:35:58.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:35:58.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.453+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3418142520 con 0x7f34180a98f0 2024-03-25T18:35:58.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.454+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501772536 0 0) 0x7f34180b28a0 con 0x7f34181430c0 2024-03-25T18:35:58.455 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.454+0000 7f341d160640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.454+0000 7f341d160640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.454+0000 7f341d160640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.454+0000 7f341d160640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:35:58.456 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.454+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34180b2a30 con 0x7f34181430c0 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.460+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1145029233 0 0) 0x7f3418142520 con 0x7f34180a98f0 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.460+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 >> v1:172.21.15.133:6789/0 conn(0x7f34181430c0 legacy=0x7f341814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.460+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 >> v1:172.21.15.55:6789/0 conn(0x7f341813f710 legacy=0x7f34181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.460+0000 7f341d160640 10 monclient: _finish_hunting 0 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.460+0000 7f341d160640 1 monclient: found mon.noname-b 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180b44c0 con 0x7f34180a98f0 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 10 monclient: _finish_auth 0 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 10 monclient: _check_auth_tickets 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:58.462 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3414001670 con 0x7f34180a98f0 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f34203c6640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f34203c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f341d160640 10 monclient: dump: 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:58.463 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.461+0000 7f34203c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:58.468 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34140017d0 con 0x7f34180a98f0 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f341d160640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f341d160640 1 -- 172.21.15.55:0/440219791 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3414001be0 con 0x7f34180a98f0 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f34203c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f34203c6640 4 set_mon_vals no callback set 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f341d160640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f341d160640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f341d160640 10 monclient: dump: 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:58.469 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:58.470 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:58.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f34203c6640 10 monclient: shutdown 2024-03-25T18:35:58.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f34203c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:58.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.468+0000 7f34203c6640 1 -- 172.21.15.55:0/440219791 >> v1:172.21.15.55:6790/0 conn(0x7f34180a98f0 legacy=0x7f34180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.469+0000 7f34203c6640 1 -- 172.21.15.55:0/440219791 shutdown_connections 2024-03-25T18:35:58.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.469+0000 7f34203c6640 1 -- 172.21.15.55:0/440219791 >> 172.21.15.55:0/440219791 conn(0x7f34180b7ea0 msgr2=0x7f34180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:58.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.469+0000 7f34203c6640 1 -- 172.21.15.55:0/440219791 shutdown_connections 2024-03-25T18:35:58.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.469+0000 7f34203c6640 1 -- 172.21.15.55:0/440219791 wait complete. 2024-03-25T18:35:58.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.470+0000 7f34203c6640 10 monclient: build_initial_monmap 2024-03-25T18:35:58.471 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.470+0000 7f34203c6640 10 monclient: monmap: 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.470+0000 7f34203c6640 1 librados: starting msgr at 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.470+0000 7f34203c6640 1 librados: starting objecter 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.470+0000 7f34203c6640 1 Processor -- start 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.470+0000 7f34203c6640 1 -- start start 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.470+0000 7f34203c6640 1 librados: setting wanted keys 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.470+0000 7f34203c6640 1 librados: calling monclient init 2024-03-25T18:35:58.472 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.470+0000 7f34203c6640 10 monclient: init 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 10 monclient(hunting): picked mon.noname-c con 0x7f34180a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 10 monclient(hunting): picked mon.noname-b con 0x7f341813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 10 monclient(hunting): picked mon.noname-a con 0x7f34181430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34181dc070 con 0x7f34181430c0 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3418138060 con 0x7f341813f710 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3418136e90 con 0x7f34180a98f0 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:58.474 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.471+0000 7f34203c6640 10 monclient(hunting): authenticate will time out at 7762.729171s 2024-03-25T18:35:58.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.487+0000 7f341e963640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f34181430c0 0x7f34181db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:46956/0 (socket says 172.21.15.55:46956) 2024-03-25T18:35:58.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.487+0000 7f341e963640 1 -- 172.21.15.55:0/937829050 learned_addr learned my addr 172.21.15.55:0/937829050 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:35:58.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.489+0000 7f340effd640 10 client.?.objecter ms_handle_connect 0x7f34181430c0 2024-03-25T18:35:58.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.489+0000 7f340effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:58.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.491+0000 7f340effd640 10 client.?.objecter ms_handle_connect 0x7f341813f710 2024-03-25T18:35:58.492 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.491+0000 7f340effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:58.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.499+0000 7f340effd640 10 client.?.objecter ms_handle_connect 0x7f34180a98f0 2024-03-25T18:35:58.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.499+0000 7f340effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:58.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.501+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521537874 0 0) 0x7f34181dc070 con 0x7f34181430c0 2024-03-25T18:35:58.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.501+0000 7f340effd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.501+0000 7f340effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.501+0000 7f340effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.501+0000 7f340effd640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:35:58.502 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.501+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc004eb0 con 0x7f34181430c0 2024-03-25T18:35:58.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.507+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428890161 0 0) 0x7f3418138060 con 0x7f341813f710 2024-03-25T18:35:58.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.507+0000 7f340effd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.507+0000 7f340effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.507+0000 7f340effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.507+0000 7f340effd640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:35:58.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.507+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34181dc070 con 0x7f341813f710 2024-03-25T18:35:58.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.516+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3042997554 0 0) 0x7f33fc004eb0 con 0x7f34181430c0 2024-03-25T18:35:58.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3418138060 con 0x7f34181430c0 2024-03-25T18:35:58.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3410001bd0 con 0x7f34181430c0 2024-03-25T18:35:58.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:58.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 10 monclient(hunting): dump: 2024-03-25T18:35:58.518 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:58.518 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:58.518 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249139528 0 0) 0x7f3418136e90 con 0x7f34180a98f0 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:35:58.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.517+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc004eb0 con 0x7f34180a98f0 2024-03-25T18:35:58.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.530+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3309508681 0 0) 0x7f34181dc070 con 0x7f341813f710 2024-03-25T18:35:58.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.530+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3418136e90 con 0x7f341813f710 2024-03-25T18:35:58.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.530+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3408001950 con 0x7f341813f710 2024-03-25T18:35:58.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.530+0000 7f340effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.530+0000 7f340effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:58.531 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.530+0000 7f340effd640 10 monclient(hunting): dump: 2024-03-25T18:35:58.531 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:58.531 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:58.531 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.532 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.532 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.532 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:58.532 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:58.532 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:58.532 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:58.532 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:58.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2529745401 0 0) 0x7f3418138060 con 0x7f34181430c0 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 >> v1:172.21.15.55:6790/0 conn(0x7f34180a98f0 legacy=0x7f3418137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 >> v1:172.21.15.133:6789/0 conn(0x7f341813f710 legacy=0x7f34181db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 1 monclient: found mon.a 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3418137170 con 0x7f34181430c0 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 10 monclient: _finish_auth 0 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f340effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f34203c6640 5 monclient: authenticate success, global_id 4587 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f34203c6640 10 monclient: _renew_subs 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f34203c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:58.536 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341813cbb0 con 0x7f34181430c0 2024-03-25T18:35:58.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f34203c6640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:58.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f34203c6640 10 monclient: _renew_subs 2024-03-25T18:35:58.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f34203c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:58.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34181361c0 con 0x7f34181430c0 2024-03-25T18:35:58.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.534+0000 7f34203c6640 1 librados: init done 2024-03-25T18:35:58.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.535+0000 7f34203c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:58.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.535+0000 7f34203c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:58.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.535+0000 7f34203c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:58.537 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.535+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34180b3ef0 con 0x7f34181430c0 2024-03-25T18:35:58.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.543+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3410002080 con 0x7f34181430c0 2024-03-25T18:35:58.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.543+0000 7f340effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:58.544 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.543+0000 7f341c95f640 4 set_mon_vals no callback set 2024-03-25T18:35:58.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.856+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3410004d00 con 0x7f34181430c0 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.856+0000 7f340effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.856+0000 7f340effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.856+0000 7f340effd640 10 monclient: dump: 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:35:58.857 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:35:58.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.856+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f3410004ea0 con 0x7f34181430c0 2024-03-25T18:35:58.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.857+0000 7f341e162640 10 monclient: get_auth_request con 0x7f33fc06dec0 auth_method 0 2024-03-25T18:35:58.858 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.857+0000 7f340effd640 10 client.4587.objecter ms_handle_connect 0x7f33fc06dec0 2024-03-25T18:35:58.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.863+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f3410054b60 con 0x7f34181430c0 2024-03-25T18:35:58.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.863+0000 7f340effd640 10 client.4587.objecter ms_dispatch 0x7f3418000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:58.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.863+0000 7f340effd640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:58.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.863+0000 7f340effd640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:58.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.863+0000 7f340effd640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T18:35:58.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.863+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34100504a0 con 0x7f34181430c0 2024-03-25T18:35:58.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.864+0000 7f340effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:58.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:58.864+0000 7f340effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:59.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.118+0000 7f34203c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:59.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.118+0000 7f34203c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:59.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.118+0000 7f34203c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:35:59.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.118+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34180627b0 con 0x7f34181430c0 2024-03-25T18:35:59.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.156+0000 7f340effd640 1 -- 172.21.15.55:0/937829050 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f34100525a0 con 0x7f34181430c0 2024-03-25T18:35:59.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.156+0000 7f340effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:59.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.156+0000 7f340effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:59.157 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:35:59.158 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:59.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 10 librados: watch_flush enter 2024-03-25T18:35:59.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 10 librados: watch_flush exit 2024-03-25T18:35:59.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T18:35:59.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 10 client.4587.objecter successfully canceled tick 2024-03-25T18:35:59.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 >> v1:172.21.15.55:6800/286536064 conn(0x7f33fc06dec0 legacy=0x7f33fc070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:59.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 10 monclient: shutdown 2024-03-25T18:35:59.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:59.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 >> v1:172.21.15.55:6789/0 conn(0x7f34181430c0 legacy=0x7f34181db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:59.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 shutdown_connections 2024-03-25T18:35:59.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 >> 172.21.15.55:0/937829050 conn(0x7f34180b7ea0 msgr2=0x7f3418074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:59.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 shutdown_connections 2024-03-25T18:35:59.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 1 -- 172.21.15.55:0/937829050 wait complete. 2024-03-25T18:35:59.161 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:35:59.159+0000 7f34203c6640 1 librados: shutdown 2024-03-25T18:35:59.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:05.979 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.079+0000 7fd74d79a640 10 monclient: get_monmap_and_config 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.079+0000 7fd74d79a640 10 monclient: build_initial_monmap 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.079+0000 7fd74d79a640 10 monclient: monmap: 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:36:06.081071+0000 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:36:06.081071+0000 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:06.081 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:36:06.082 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:36:06.082 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.079+0000 7fd74d79a640 1 Processor -- start 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.079+0000 7fd74d79a640 1 -- start start 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.079+0000 7fd74d79a640 10 monclient: init 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 10 monclient(hunting): picked mon.noname-c con 0x7fd7480b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 10 monclient(hunting): picked mon.noname-b con 0x7fd7480b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 10 monclient(hunting): picked mon.noname-a con 0x7fd74814be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7480af940 con 0x7fd74814be90 2024-03-25T18:36:06.083 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7480b0a70 con 0x7fd7480b4d60 2024-03-25T18:36:06.084 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7480b0d50 con 0x7fd7480b1730 2024-03-25T18:36:06.084 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:06.084 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.080+0000 7fd74d79a640 10 monclient(hunting): authenticate will time out at 7770.338896s 2024-03-25T18:36:06.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.084+0000 7fd746ffd640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fd7480b4d60 0x7fd74813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:59800/0 (socket says 172.21.15.55:59800) 2024-03-25T18:36:06.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.084+0000 7fd746ffd640 1 -- 172.21.15.55:0/2317219320 learned_addr learned my addr 172.21.15.55:0/2317219320 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:06.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.096+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376668866 0 0) 0x7fd7480b0a70 con 0x7fd7480b4d60 2024-03-25T18:36:06.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.096+0000 7fd745ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:06.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.096+0000 7fd745ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:06.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.096+0000 7fd745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:06.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.096+0000 7fd745ffb640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:36:06.098 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.096+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd730003520 con 0x7fd7480b4d60 2024-03-25T18:36:06.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.109+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540869353 0 0) 0x7fd7480af940 con 0x7fd74814be90 2024-03-25T18:36:06.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.109+0000 7fd745ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:06.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.109+0000 7fd745ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:06.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.109+0000 7fd745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:06.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.109+0000 7fd745ffb640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:36:06.111 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.109+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7480b0a70 con 0x7fd74814be90 2024-03-25T18:36:06.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 837514408 0 0) 0x7fd730003520 con 0x7fd7480b4d60 2024-03-25T18:36:06.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 >> v1:172.21.15.55:6790/0 conn(0x7fd7480b1730 legacy=0x7fd7480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 >> v1:172.21.15.55:6789/0 conn(0x7fd74814be90 legacy=0x7fd74814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:36:06.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 1 monclient: found mon.noname-c 2024-03-25T18:36:06.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:06.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:36:06.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7480b1030 con 0x7fd7480b4d60 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 10 monclient: _finish_auth 0 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd738001690 con 0x7fd7480b4d60 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd745ffb640 10 monclient: dump: 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:06.125 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:06.126 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:06.126 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:06.126 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:06.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd74d79a640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:36:06.126 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.122+0000 7fd74d79a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7380017f0 con 0x7fd7480b4d60 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd745ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd745ffb640 1 -- 172.21.15.55:0/2317219320 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd738001c00 con 0x7fd7480b4d60 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd74d79a640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd74d79a640 4 set_mon_vals no callback set 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd745ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd745ffb640 10 monclient: dump: 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:06.136 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.137 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.137 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.137 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:06.137 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:06.137 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:06.137 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:06.137 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:06.139 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd74d79a640 10 monclient: shutdown 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd74d79a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.134+0000 7fd74d79a640 1 -- 172.21.15.55:0/2317219320 >> v1:172.21.15.133:6789/0 conn(0x7fd7480b4d60 legacy=0x7fd74813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.135+0000 7fd74d79a640 1 -- 172.21.15.55:0/2317219320 shutdown_connections 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.135+0000 7fd74d79a640 1 -- 172.21.15.55:0/2317219320 >> 172.21.15.55:0/2317219320 conn(0x7fd7480617a0 msgr2=0x7fd748061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.135+0000 7fd74d79a640 1 -- 172.21.15.55:0/2317219320 shutdown_connections 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.135+0000 7fd74d79a640 1 -- 172.21.15.55:0/2317219320 wait complete. 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.135+0000 7fd74d79a640 10 monclient: build_initial_monmap 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.135+0000 7fd74d79a640 10 monclient: monmap: 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:06.140 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 1 librados: starting msgr at 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 1 librados: starting objecter 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 1 Processor -- start 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 1 -- start start 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 1 librados: setting wanted keys 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 1 librados: calling monclient init 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 10 monclient: init 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 10 monclient(hunting): picked mon.noname-a con 0x7fd7480b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 10 monclient(hunting): picked mon.noname-b con 0x7fd7480b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 10 monclient(hunting): picked mon.noname-c con 0x7fd74814be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:06.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd748141ce0 con 0x7fd7480b1730 2024-03-25T18:36:06.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.136+0000 7fd74d79a640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd748142610 con 0x7fd7480b4d60 2024-03-25T18:36:06.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.137+0000 7fd74d79a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7481dae80 con 0x7fd74814be90 2024-03-25T18:36:06.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.137+0000 7fd74d79a640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:06.142 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.137+0000 7fd74d79a640 10 monclient(hunting): authenticate will time out at 7770.395371s 2024-03-25T18:36:06.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.152+0000 7fd7467fc640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fd7480b4d60 0x7fd7481db430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:59824/0 (socket says 172.21.15.55:59824) 2024-03-25T18:36:06.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.152+0000 7fd7467fc640 1 -- 172.21.15.55:0/2806076720 learned_addr learned my addr 172.21.15.55:0/2806076720 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:06.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.158+0000 7fd72f7fe640 10 client.?.objecter ms_handle_connect 0x7fd7480b4d60 2024-03-25T18:36:06.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.158+0000 7fd72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:06.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.159+0000 7fd72f7fe640 10 client.?.objecter ms_handle_connect 0x7fd7480b1730 2024-03-25T18:36:06.160 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.159+0000 7fd72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:06.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.164+0000 7fd72f7fe640 10 client.?.objecter ms_handle_connect 0x7fd74814be90 2024-03-25T18:36:06.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.164+0000 7fd72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:06.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.168+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244804142 0 0) 0x7fd748142610 con 0x7fd7480b4d60 2024-03-25T18:36:06.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.168+0000 7fd72f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:06.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.168+0000 7fd72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:06.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.168+0000 7fd72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:06.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.168+0000 7fd72f7fe640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:36:06.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.168+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd734004eb0 con 0x7fd7480b4d60 2024-03-25T18:36:06.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.170+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35877953 0 0) 0x7fd748141ce0 con 0x7fd7480b1730 2024-03-25T18:36:06.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.170+0000 7fd72f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:06.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.170+0000 7fd72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:06.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.170+0000 7fd72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:06.172 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.170+0000 7fd72f7fe640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:36:06.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.170+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd748142610 con 0x7fd7480b1730 2024-03-25T18:36:06.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.175+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597530866 0 0) 0x7fd7481dae80 con 0x7fd74814be90 2024-03-25T18:36:06.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.175+0000 7fd72f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:06.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.175+0000 7fd72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:06.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.175+0000 7fd72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:06.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.175+0000 7fd72f7fe640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:36:06.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.175+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd748141ce0 con 0x7fd74814be90 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.192+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 287502278 0 0) 0x7fd748141ce0 con 0x7fd74814be90 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.192+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7481dae80 con 0x7fd74814be90 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.192+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd740001bd0 con 0x7fd74814be90 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.192+0000 7fd72f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.192+0000 7fd72f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:06.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.192+0000 7fd72f7fe640 10 monclient(hunting): dump: 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:06.195 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.192+0000 7fd72f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:06.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.200+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4252817561 0 0) 0x7fd734004eb0 con 0x7fd7480b4d60 2024-03-25T18:36:06.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.200+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd748141ce0 con 0x7fd7480b4d60 2024-03-25T18:36:06.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.200+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd73c001a30 con 0x7fd7480b4d60 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.200+0000 7fd72f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.200+0000 7fd72f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.200+0000 7fd72f7fe640 10 monclient(hunting): dump: 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:06.203 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.209+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1945413371 0 0) 0x7fd748142610 con 0x7fd7480b1730 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.209+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd734004eb0 con 0x7fd7480b1730 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.209+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd738001b00 con 0x7fd7480b1730 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.209+0000 7fd72f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.209+0000 7fd72f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.209+0000 7fd72f7fe640 10 monclient(hunting): dump: 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:06.211 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:06.212 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:06.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.221+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3604951087 0 0) 0x7fd7481dae80 con 0x7fd74814be90 2024-03-25T18:36:06.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 >> v1:172.21.15.133:6789/0 conn(0x7fd7480b4d60 legacy=0x7fd7481db430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 >> v1:172.21.15.55:6789/0 conn(0x7fd7480b1730 legacy=0x7fd748142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:06.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 1 monclient: found mon.c 2024-03-25T18:36:06.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:06.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:06.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7481db010 con 0x7fd74814be90 2024-03-25T18:36:06.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd72f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd74d79a640 5 monclient: authenticate success, global_id 4577 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd74d79a640 10 monclient: _renew_subs 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd74d79a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd748143240 con 0x7fd74814be90 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd74d79a640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd74d79a640 10 monclient: _renew_subs 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd74d79a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74813bd20 con 0x7fd74814be90 2024-03-25T18:36:06.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.222+0000 7fd74d79a640 1 librados: init done 2024-03-25T18:36:06.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.223+0000 7fd74d79a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:06.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.223+0000 7fd74d79a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:06.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.223+0000 7fd74d79a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:06.226 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.223+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7480627b0 con 0x7fd74814be90 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.585+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd740001fe0 con 0x7fd74814be90 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.585+0000 7fd72f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.585+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fd740003b10 con 0x7fd74814be90 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.585+0000 7fd72f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.585+0000 7fd72f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.585+0000 7fd72f7fe640 10 monclient: dump: 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.587 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:06.588 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:06.588 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:06.588 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:06.588 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:06.588 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:06.588 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:06.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.585+0000 7fd7457fa640 4 set_mon_vals no callback set 2024-03-25T18:36:06.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.585+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fd740019ae0 con 0x7fd74814be90 2024-03-25T18:36:06.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.586+0000 7fd746ffd640 10 monclient: get_auth_request con 0x7fd73406e0a0 auth_method 0 2024-03-25T18:36:06.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.586+0000 7fd72f7fe640 10 client.4577.objecter ms_handle_connect 0x7fd73406e0a0 2024-03-25T18:36:06.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.592+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fd740053730 con 0x7fd74814be90 2024-03-25T18:36:06.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.592+0000 7fd72f7fe640 10 client.4577.objecter ms_dispatch 0x7fd748000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:06.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.592+0000 7fd72f7fe640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:06.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.592+0000 7fd72f7fe640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:06.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.592+0000 7fd72f7fe640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-25T18:36:06.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.592+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd74004f070 con 0x7fd74814be90 2024-03-25T18:36:06.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.592+0000 7fd72f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:06.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.592+0000 7fd72f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:06.844 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.842+0000 7fd74d79a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:06.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.842+0000 7fd74d79a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:06.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.842+0000 7fd74d79a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:06.845 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.842+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd748137cc0 con 0x7fd74814be90 2024-03-25T18:36:06.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.920+0000 7fd72f7fe640 1 -- 172.21.15.55:0/2806076720 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fd740051170 con 0x7fd74814be90 2024-03-25T18:36:06.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.920+0000 7fd72f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:06.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.920+0000 7fd72f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:06.922 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:36:06.922 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:06.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 10 librados: watch_flush enter 2024-03-25T18:36:06.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 10 librados: watch_flush exit 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 10 client.4577.objecter successfully canceled tick 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 >> v1:172.21.15.55:6800/286536064 conn(0x7fd73406e0a0 legacy=0x7fd734070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 10 monclient: shutdown 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 >> v1:172.21.15.55:6790/0 conn(0x7fd74814be90 legacy=0x7fd748140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 shutdown_connections 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 >> 172.21.15.55:0/2806076720 conn(0x7fd7480617a0 msgr2=0x7fd7480b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 shutdown_connections 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 1 -- 172.21.15.55:0/2806076720 wait complete. 2024-03-25T18:36:06.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:06.922+0000 7fd74d79a640 1 librados: shutdown 2024-03-25T18:36:06.937 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:13.744 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.863+0000 7f9b7c0c8640 10 monclient: get_monmap_and_config 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.863+0000 7f9b7c0c8640 10 monclient: build_initial_monmap 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.863+0000 7f9b7c0c8640 10 monclient: monmap: 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:36:13.864689+0000 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:36:13.864689+0000 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:36:13.865 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:13.866 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.864+0000 7f9b7c0c8640 1 Processor -- start 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.864+0000 7f9b7c0c8640 1 -- start start 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.864+0000 7f9b7c0c8640 10 monclient: init 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b740a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b741430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b74142520 con 0x7f9b741430c0 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b740b28a0 con 0x7f9b7413f710 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b740b2a30 con 0x7f9b740a98f0 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:13.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.865+0000 7f9b7c0c8640 10 monclient(hunting): authenticate will time out at 7778.122697s 2024-03-25T18:36:13.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.870+0000 7f9b79663640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f9b7413f710 0x7f9b741429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:50904/0 (socket says 172.21.15.55:50904) 2024-03-25T18:36:13.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.870+0000 7f9b79663640 1 -- 172.21.15.55:0/1569997396 learned_addr learned my addr 172.21.15.55:0/1569997396 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:13.883 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.882+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023209347 0 0) 0x7f9b740b2a30 con 0x7f9b740a98f0 2024-03-25T18:36:13.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.883+0000 7f9b78e62640 10 monclient(hunting): _negotiate 2024-03-25T18:36:13.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.883+0000 7f9b78e62640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:13.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.883+0000 7f9b78e62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:13.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.883+0000 7f9b78e62640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:36:13.884 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.883+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b60003520 con 0x7f9b740a98f0 2024-03-25T18:36:13.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.889+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758878555 0 0) 0x7f9b740b28a0 con 0x7f9b7413f710 2024-03-25T18:36:13.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.889+0000 7f9b78e62640 10 monclient(hunting): _negotiate 2024-03-25T18:36:13.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.889+0000 7f9b78e62640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:13.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.889+0000 7f9b78e62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:13.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.889+0000 7f9b78e62640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:36:13.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.889+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b740b2a30 con 0x7f9b7413f710 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2253840371 0 0) 0x7f9b60003520 con 0x7f9b740a98f0 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 >> v1:172.21.15.133:6789/0 conn(0x7f9b7413f710 legacy=0x7f9b741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 >> v1:172.21.15.55:6789/0 conn(0x7f9b741430c0 legacy=0x7f9b7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 10 monclient: _finish_hunting 0 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 1 monclient: found mon.noname-b 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b740b44c0 con 0x7f9b740a98f0 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 10 monclient: _finish_auth 0 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 10 monclient: _check_auth_tickets 2024-03-25T18:36:13.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9b70001690 con 0x7f9b740a98f0 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b7c0c8640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b7c0c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b78e62640 10 monclient: dump: 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:13.897 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:13.898 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.894+0000 7f9b7c0c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:13.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.914+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b700017f0 con 0x7f9b740a98f0 2024-03-25T18:36:13.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.914+0000 7f9b78e62640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.914+0000 7f9b78e62640 1 -- 172.21.15.55:0/1569997396 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9b70001c00 con 0x7f9b740a98f0 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.914+0000 7f9b78e62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.914+0000 7f9b78e62640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.914+0000 7f9b78e62640 10 monclient: dump: 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:13.916 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.914+0000 7f9b7c0c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:13.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.914+0000 7f9b7c0c8640 4 set_mon_vals no callback set 2024-03-25T18:36:13.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.915+0000 7f9b7c0c8640 10 monclient: shutdown 2024-03-25T18:36:13.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.915+0000 7f9b7c0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:13.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.915+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/1569997396 >> v1:172.21.15.55:6790/0 conn(0x7f9b740a98f0 legacy=0x7f9b740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:13.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.915+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/1569997396 shutdown_connections 2024-03-25T18:36:13.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.915+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/1569997396 >> 172.21.15.55:0/1569997396 conn(0x7f9b740b7ea0 msgr2=0x7f9b740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:13.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.915+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/1569997396 shutdown_connections 2024-03-25T18:36:13.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.915+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/1569997396 wait complete. 2024-03-25T18:36:13.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.916+0000 7f9b7c0c8640 10 monclient: build_initial_monmap 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.916+0000 7f9b7c0c8640 10 monclient: monmap: 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.916+0000 7f9b7c0c8640 1 librados: starting msgr at 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.916+0000 7f9b7c0c8640 1 librados: starting objecter 2024-03-25T18:36:13.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.916+0000 7f9b7c0c8640 1 Processor -- start 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 1 -- start start 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 1 librados: setting wanted keys 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 1 librados: calling monclient init 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 10 monclient: init 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f9b740a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7413f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b741430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b74137720 con 0x7f9b7413f710 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b741dc070 con 0x7f9b741430c0 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b74139140 con 0x7f9b740a98f0 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:13.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.917+0000 7f9b7c0c8640 10 monclient(hunting): authenticate will time out at 7778.175197s 2024-03-25T18:36:13.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.926+0000 7f9b7a665640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f9b741430c0 0x7f9b741db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:50918/0 (socket says 172.21.15.55:50918) 2024-03-25T18:36:13.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.926+0000 7f9b7a665640 1 -- 172.21.15.55:0/4258733257 learned_addr learned my addr 172.21.15.55:0/4258733257 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:13.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.931+0000 7f9b6a7fc640 10 client.?.objecter ms_handle_connect 0x7f9b740a98f0 2024-03-25T18:36:13.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.931+0000 7f9b6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:13.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.933+0000 7f9b6a7fc640 10 client.?.objecter ms_handle_connect 0x7f9b741430c0 2024-03-25T18:36:13.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.933+0000 7f9b6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:13.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.941+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004809544 0 0) 0x7f9b741dc070 con 0x7f9b741430c0 2024-03-25T18:36:13.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.941+0000 7f9b6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:13.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.941+0000 7f9b6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:13.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.941+0000 7f9b6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:13.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.941+0000 7f9b6a7fc640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:36:13.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.941+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b58004eb0 con 0x7f9b741430c0 2024-03-25T18:36:13.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.943+0000 7f9b6a7fc640 10 client.?.objecter ms_handle_connect 0x7f9b7413f710 2024-03-25T18:36:13.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.943+0000 7f9b6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:13.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.955+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 652765746 0 0) 0x7f9b58004eb0 con 0x7f9b741430c0 2024-03-25T18:36:13.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.955+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b741dc070 con 0x7f9b741430c0 2024-03-25T18:36:13.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.955+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9b6c001c90 con 0x7f9b741430c0 2024-03-25T18:36:13.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.955+0000 7f9b6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:13.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.955+0000 7f9b6a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:13.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.955+0000 7f9b6a7fc640 10 monclient(hunting): dump: 2024-03-25T18:36:13.956 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:13.956 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:13.956 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:13.957 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:13.957 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:13.957 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:13.957 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:13.957 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:13.957 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:13.957 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:13.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.955+0000 7f9b6a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:13.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.957+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348354939 0 0) 0x7f9b74137720 con 0x7f9b7413f710 2024-03-25T18:36:13.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.957+0000 7f9b6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:13.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.957+0000 7f9b6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:13.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.957+0000 7f9b6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:13.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.957+0000 7f9b6a7fc640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:36:13.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.957+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b58004eb0 con 0x7f9b7413f710 2024-03-25T18:36:13.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.959+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181307399 0 0) 0x7f9b74139140 con 0x7f9b740a98f0 2024-03-25T18:36:13.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.959+0000 7f9b6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:13.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.959+0000 7f9b6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:13.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.959+0000 7f9b6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:13.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.959+0000 7f9b6a7fc640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:36:13.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.959+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b74137720 con 0x7f9b740a98f0 2024-03-25T18:36:13.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2148322652 0 0) 0x7f9b741dc070 con 0x7f9b741430c0 2024-03-25T18:36:13.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 >> v1:172.21.15.55:6790/0 conn(0x7f9b740a98f0 legacy=0x7f9b74137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:13.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 >> v1:172.21.15.55:6789/0 conn(0x7f9b7413f710 legacy=0x7f9b741db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:13.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:13.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 1 monclient: found mon.b 2024-03-25T18:36:13.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:13.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:13.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b74139420 con 0x7f9b741430c0 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b7c0c8640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b7c0c8640 10 monclient: _renew_subs 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b7c0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b74136dc0 con 0x7f9b741430c0 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b7c0c8640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b7c0c8640 10 monclient: _renew_subs 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b7c0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b74139840 con 0x7f9b741430c0 2024-03-25T18:36:13.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.980+0000 7f9b7c0c8640 1 librados: init done 2024-03-25T18:36:13.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.982+0000 7f9b7c0c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:13.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.982+0000 7f9b7c0c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:13.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.982+0000 7f9b7c0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:13.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:13.982+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7414f2d0 con 0x7f9b741430c0 2024-03-25T18:36:14.282 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.281+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b6c003990 con 0x7f9b741430c0 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.281+0000 7f9b6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.281+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9b6c003da0 con 0x7f9b741430c0 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.281+0000 7f9b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.281+0000 7f9b6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.281+0000 7f9b6a7fc640 10 monclient: dump: 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:14.283 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:14.284 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:14.284 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:14.284 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:14.284 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:14.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.281+0000 7f9b6bfff640 4 set_mon_vals no callback set 2024-03-25T18:36:14.284 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.282+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f9b6c003fc0 con 0x7f9b741430c0 2024-03-25T18:36:14.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.283+0000 7f9b79e64640 10 monclient: get_auth_request con 0x7f9b5806de60 auth_method 0 2024-03-25T18:36:14.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.283+0000 7f9b6a7fc640 10 client.4570.objecter ms_handle_connect 0x7f9b5806de60 2024-03-25T18:36:14.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.287+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f9b6c054b10 con 0x7f9b741430c0 2024-03-25T18:36:14.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.287+0000 7f9b6a7fc640 10 client.4570.objecter ms_dispatch 0x7f9b74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:14.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.287+0000 7f9b6a7fc640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:14.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.287+0000 7f9b6a7fc640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:14.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.287+0000 7f9b6a7fc640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:36:14.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.287+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b6c050620 con 0x7f9b741430c0 2024-03-25T18:36:14.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.287+0000 7f9b6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:14.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.287+0000 7f9b6a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:14.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.555+0000 7f9b7c0c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:14.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.555+0000 7f9b7c0c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:14.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.555+0000 7f9b7c0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:14.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.555+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b74136f50 con 0x7f9b741430c0 2024-03-25T18:36:14.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.647+0000 7f9b6a7fc640 1 -- 172.21.15.55:0/4258733257 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f9b6c052550 con 0x7f9b741430c0 2024-03-25T18:36:14.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.647+0000 7f9b6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:14.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.647+0000 7f9b6a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:14.648 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:36:14.649 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:14.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 10 librados: watch_flush enter 2024-03-25T18:36:14.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 10 librados: watch_flush exit 2024-03-25T18:36:14.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 >> v1:172.21.15.55:6800/286536064 conn(0x7f9b5806de60 legacy=0x7f9b58070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 10 monclient: shutdown 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 >> v1:172.21.15.133:6789/0 conn(0x7f9b741430c0 legacy=0x7f9b741db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 shutdown_connections 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 >> 172.21.15.55:0/4258733257 conn(0x7f9b740b7ea0 msgr2=0x7f9b74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 shutdown_connections 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 1 -- 172.21.15.55:0/4258733257 wait complete. 2024-03-25T18:36:14.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:14.650+0000 7f9b7c0c8640 1 librados: shutdown 2024-03-25T18:36:14.662 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:16.415 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:08:26.584 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:16.470 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:08:26.639 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:16.511 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:08:26.681 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:16.517 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:08:26.687 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:16.526 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:26.663 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:16.527 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:08:26.697 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:16.534 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:26.671 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:16.551 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:08:26.721 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:16.565 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:26.702 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:21.469 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:21.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.583+0000 7f8fab072640 10 monclient: get_monmap_and_config 2024-03-25T18:36:21.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.583+0000 7f8fab072640 10 monclient: build_initial_monmap 2024-03-25T18:36:21.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.583+0000 7f8fab072640 10 monclient: monmap: 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:36:21.584326+0000 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:36:21.584326+0000 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:36:21.585 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:21.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.584+0000 7f8fab072640 1 Processor -- start 2024-03-25T18:36:21.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.584+0000 7f8fab072640 1 -- start start 2024-03-25T18:36:21.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.584+0000 7f8fab072640 10 monclient: init 2024-03-25T18:36:21.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:21.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:21.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa40a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:21.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:21.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa41430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:21.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa4142520 con 0x7f8fa40a98f0 2024-03-25T18:36:21.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa40b28a0 con 0x7f8fa413f710 2024-03-25T18:36:21.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa40b2a30 con 0x7f8fa41430c0 2024-03-25T18:36:21.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:21.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.585+0000 7f8fab072640 10 monclient(hunting): authenticate will time out at 7785.842633s 2024-03-25T18:36:21.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.594+0000 7f8fa960f640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f8fa41430c0 0x7f8fa414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:36052/0 (socket says 172.21.15.55:36052) 2024-03-25T18:36:21.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.594+0000 7f8fa960f640 1 -- 172.21.15.55:0/3190710597 learned_addr learned my addr 172.21.15.55:0/3190710597 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:21.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.607+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085612022 0 0) 0x7f8fa40b28a0 con 0x7f8fa413f710 2024-03-25T18:36:21.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.607+0000 7f8fa37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:21.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.607+0000 7f8fa37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:21.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.607+0000 7f8fa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:21.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.607+0000 7f8fa37fe640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:36:21.609 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.607+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f94003520 con 0x7f8fa413f710 2024-03-25T18:36:21.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.617+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758216767 0 0) 0x7f8fa40b2a30 con 0x7f8fa41430c0 2024-03-25T18:36:21.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.617+0000 7f8fa37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:21.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.617+0000 7f8fa37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:21.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.617+0000 7f8fa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:21.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.617+0000 7f8fa37fe640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:36:21.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.617+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa40b28a0 con 0x7f8fa41430c0 2024-03-25T18:36:21.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043250209 0 0) 0x7f8f94003520 con 0x7f8fa413f710 2024-03-25T18:36:21.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 >> v1:172.21.15.55:6790/0 conn(0x7f8fa41430c0 legacy=0x7f8fa414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 >> v1:172.21.15.55:6789/0 conn(0x7f8fa40a98f0 legacy=0x7f8fa40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 1 monclient: found mon.noname-c 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa40b44c0 con 0x7f8fa413f710 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8f90001690 con 0x7f8fa413f710 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fa37fe640 10 monclient: dump: 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:21.624 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.625 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.625 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.625 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:21.625 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:21.625 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:21.625 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:21.625 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:21.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fab072640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:36:21.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.622+0000 7f8fab072640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:21.632 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f900017f0 con 0x7f8fa413f710 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fa37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fa37fe640 1 -- 172.21.15.55:0/3190710597 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8f90001c00 con 0x7f8fa413f710 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fab072640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fab072640 4 set_mon_vals no callback set 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fa37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fa37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fa37fe640 10 monclient: dump: 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.633 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fab072640 10 monclient: shutdown 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fab072640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fab072640 1 -- 172.21.15.55:0/3190710597 >> v1:172.21.15.133:6789/0 conn(0x7f8fa413f710 legacy=0x7f8fa41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fab072640 1 -- 172.21.15.55:0/3190710597 shutdown_connections 2024-03-25T18:36:21.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fab072640 1 -- 172.21.15.55:0/3190710597 >> 172.21.15.55:0/3190710597 conn(0x7f8fa40b7ea0 msgr2=0x7f8fa40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:21.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fab072640 1 -- 172.21.15.55:0/3190710597 shutdown_connections 2024-03-25T18:36:21.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.631+0000 7f8fab072640 1 -- 172.21.15.55:0/3190710597 wait complete. 2024-03-25T18:36:21.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.632+0000 7f8fab072640 10 monclient: build_initial_monmap 2024-03-25T18:36:21.636 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.632+0000 7f8fab072640 10 monclient: monmap: 2024-03-25T18:36:21.636 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:21.636 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:21.636 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.632+0000 7f8fab072640 1 librados: starting msgr at 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.632+0000 7f8fab072640 1 librados: starting objecter 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 1 Processor -- start 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 1 -- start start 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 1 librados: setting wanted keys 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 1 librados: calling monclient init 2024-03-25T18:36:21.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 10 monclient: init 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa40a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa413f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa41430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa4139070 con 0x7f8fa41430c0 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa4133710 con 0x7f8fa40a98f0 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa41338a0 con 0x7f8fa413f710 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:21.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.633+0000 7f8fab072640 10 monclient(hunting): authenticate will time out at 7785.891291s 2024-03-25T18:36:21.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.640+0000 7f8fa8e0e640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f8fa40a98f0 0x7f8fa41378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:44392/0 (socket says 172.21.15.55:44392) 2024-03-25T18:36:21.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.640+0000 7f8fa8e0e640 1 -- 172.21.15.55:0/308507260 learned_addr learned my addr 172.21.15.55:0/308507260 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:21.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.642+0000 7f8fa17fa640 10 client.?.objecter ms_handle_connect 0x7f8fa40a98f0 2024-03-25T18:36:21.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.642+0000 7f8fa17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:21.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.654+0000 7f8fa17fa640 10 client.?.objecter ms_handle_connect 0x7f8fa413f710 2024-03-25T18:36:21.655 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.654+0000 7f8fa17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:21.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.655+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930804306 0 0) 0x7f8fa4133710 con 0x7f8fa40a98f0 2024-03-25T18:36:21.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.655+0000 7f8fa17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:21.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.655+0000 7f8fa17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:21.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.655+0000 7f8fa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:21.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.655+0000 7f8fa17fa640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:36:21.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.655+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f88004eb0 con 0x7f8fa40a98f0 2024-03-25T18:36:21.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.660+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268847396 0 0) 0x7f8fa41338a0 con 0x7f8fa413f710 2024-03-25T18:36:21.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.660+0000 7f8fa17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:21.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.660+0000 7f8fa17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:21.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.660+0000 7f8fa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:21.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.660+0000 7f8fa17fa640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:36:21.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.660+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa4133710 con 0x7f8fa413f710 2024-03-25T18:36:21.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.665+0000 7f8fa17fa640 10 client.?.objecter ms_handle_connect 0x7f8fa41430c0 2024-03-25T18:36:21.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.665+0000 7f8fa17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:21.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.681+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 613557729 0 0) 0x7f8fa4133710 con 0x7f8fa413f710 2024-03-25T18:36:21.682 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.681+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fa41338a0 con 0x7f8fa413f710 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.681+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8f90001b00 con 0x7f8fa413f710 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.681+0000 7f8fa17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.681+0000 7f8fa17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.681+0000 7f8fa17fa640 10 monclient(hunting): dump: 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:21.683 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:21.684 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:21.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.681+0000 7f8fa17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:21.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.686+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264330643 0 0) 0x7f8fa4139070 con 0x7f8fa41430c0 2024-03-25T18:36:21.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.686+0000 7f8fa17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:21.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.686+0000 7f8fa17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:21.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.686+0000 7f8fa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:21.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.686+0000 7f8fa17fa640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:36:21.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.686+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa4133710 con 0x7f8fa41430c0 2024-03-25T18:36:21.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.691+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3231867592 0 0) 0x7f8f88004eb0 con 0x7f8fa40a98f0 2024-03-25T18:36:21.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.691+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fa4139070 con 0x7f8fa40a98f0 2024-03-25T18:36:21.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.691+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8f98001a30 con 0x7f8fa40a98f0 2024-03-25T18:36:21.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.691+0000 7f8fa17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.691+0000 7f8fa17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:21.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.691+0000 7f8fa17fa640 10 monclient(hunting): dump: 2024-03-25T18:36:21.692 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:21.693 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:21.693 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.693 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.693 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.693 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:21.693 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:21.693 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:21.693 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:21.693 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:21.702 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 145589877 0 0) 0x7f8fa4139070 con 0x7f8fa40a98f0 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 >> v1:172.21.15.55:6790/0 conn(0x7f8fa413f710 legacy=0x7f8fa41daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 >> v1:172.21.15.55:6789/0 conn(0x7f8fa41430c0 legacy=0x7f8fa4133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 1 monclient: found mon.b 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa4133a30 con 0x7f8fa40a98f0 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 10 monclient: _finish_auth 0 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fa17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:21.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fab072640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fab072640 10 monclient: _renew_subs 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fab072640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa40b7100 con 0x7f8fa40a98f0 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fab072640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fab072640 10 monclient: _renew_subs 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fab072640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa4136e60 con 0x7f8fa40a98f0 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.701+0000 7f8fab072640 1 librados: init done 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.702+0000 7f8fab072640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.702+0000 7f8fab072640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:21.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.702+0000 7f8fab072640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:21.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.702+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa40627b0 con 0x7f8fa40a98f0 2024-03-25T18:36:21.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.989+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f98001c70 con 0x7f8fa40a98f0 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.989+0000 7f8fa17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.989+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8f98003a60 con 0x7f8fa40a98f0 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.989+0000 7f8fa17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.989+0000 7f8fa17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.989+0000 7f8fa17fa640 10 monclient: dump: 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:21.991 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:21.992 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:21.992 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:21.992 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:21.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.989+0000 7f8fa2ffd640 4 set_mon_vals no callback set 2024-03-25T18:36:21.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.990+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f8f98003c80 con 0x7f8fa40a98f0 2024-03-25T18:36:21.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.991+0000 7f8fa3fff640 10 monclient: get_auth_request con 0x7f8f8806df50 auth_method 0 2024-03-25T18:36:21.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.991+0000 7f8fa17fa640 10 client.4576.objecter ms_handle_connect 0x7f8f8806df50 2024-03-25T18:36:21.996 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.995+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f8f980548e0 con 0x7f8fa40a98f0 2024-03-25T18:36:21.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.995+0000 7f8fa17fa640 10 client.4576.objecter ms_dispatch 0x7f8fa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:21.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.995+0000 7f8fa17fa640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:21.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.995+0000 7f8fa17fa640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:21.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.995+0000 7f8fa17fa640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T18:36:21.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.997+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f980503f0 con 0x7f8fa40a98f0 2024-03-25T18:36:21.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.997+0000 7f8fa17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:21.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:21.997+0000 7f8fa17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:22.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.234+0000 7f8fab072640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:22.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.235+0000 7f8fab072640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:22.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.235+0000 7f8fab072640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:22.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.235+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fa4074cc0 con 0x7f8fa40a98f0 2024-03-25T18:36:22.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.259+0000 7f8fa17fa640 1 -- 172.21.15.55:0/308507260 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f8f98052320 con 0x7f8fa40a98f0 2024-03-25T18:36:22.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.259+0000 7f8fa17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:22.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.259+0000 7f8fa17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:22.260 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:36:22.261 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:22.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.261+0000 7f8fab072640 10 librados: watch_flush enter 2024-03-25T18:36:22.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.261+0000 7f8fab072640 10 librados: watch_flush exit 2024-03-25T18:36:22.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.261+0000 7f8fab072640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T18:36:22.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.261+0000 7f8fab072640 10 client.4576.objecter successfully canceled tick 2024-03-25T18:36:22.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.262+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 >> v1:172.21.15.55:6800/286536064 conn(0x7f8f8806df50 legacy=0x7f8f88070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:22.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.262+0000 7f8fab072640 10 monclient: shutdown 2024-03-25T18:36:22.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.262+0000 7f8fab072640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:22.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.262+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 >> v1:172.21.15.133:6789/0 conn(0x7f8fa40a98f0 legacy=0x7f8fa41378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:22.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.262+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 shutdown_connections 2024-03-25T18:36:22.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.262+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 >> 172.21.15.55:0/308507260 conn(0x7f8fa40b7ea0 msgr2=0x7f8fa4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:22.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.262+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 shutdown_connections 2024-03-25T18:36:22.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.262+0000 7f8fab072640 1 -- 172.21.15.55:0/308507260 wait complete. 2024-03-25T18:36:22.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:22.262+0000 7f8fab072640 1 librados: shutdown 2024-03-25T18:36:22.276 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:29.083 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:29.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.198+0000 7fabfb56b640 10 monclient: get_monmap_and_config 2024-03-25T18:36:29.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.198+0000 7fabfb56b640 10 monclient: build_initial_monmap 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.198+0000 7fabfb56b640 10 monclient: monmap: 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:36:29.199809+0000 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:36:29.199809+0000 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:36:29.200 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:29.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.199+0000 7fabfb56b640 1 Processor -- start 2024-03-25T18:36:29.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.199+0000 7fabfb56b640 1 -- start start 2024-03-25T18:36:29.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.199+0000 7fabfb56b640 10 monclient: init 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 10 monclient(hunting): picked mon.noname-c con 0x7fabf40b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 10 monclient(hunting): picked mon.noname-a con 0x7fabf40b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 10 monclient(hunting): picked mon.noname-b con 0x7fabf414be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf40af940 con 0x7fabf40b1730 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf40b0a70 con 0x7fabf40b4d60 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf40b0d50 con 0x7fabf414be90 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:29.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.200+0000 7fabfb56b640 10 monclient(hunting): authenticate will time out at 7793.457993s 2024-03-25T18:36:29.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.206+0000 7fabf9b08640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fabf414be90 0x7fabf414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:45414/0 (socket says 172.21.15.55:45414) 2024-03-25T18:36:29.207 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.206+0000 7fabf9b08640 1 -- 172.21.15.55:0/2859876465 learned_addr learned my addr 172.21.15.55:0/2859876465 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:29.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.217+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543659789 0 0) 0x7fabf40b0d50 con 0x7fabf414be90 2024-03-25T18:36:29.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.217+0000 7fabebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.217+0000 7fabebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.217+0000 7fabebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.217+0000 7fabebfff640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:36:29.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.217+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe0003520 con 0x7fabf414be90 2024-03-25T18:36:29.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.220+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163491441 0 0) 0x7fabf40b0a70 con 0x7fabf40b4d60 2024-03-25T18:36:29.221 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.220+0000 7fabebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.220+0000 7fabebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.220+0000 7fabebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.220+0000 7fabebfff640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:36:29.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.220+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf40b0d50 con 0x7fabf40b4d60 2024-03-25T18:36:29.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.223+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486227651 0 0) 0x7fabf40af940 con 0x7fabf40b1730 2024-03-25T18:36:29.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.223+0000 7fabebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.223+0000 7fabebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.223+0000 7fabebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.223+0000 7fabebfff640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:36:29.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.223+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf40b0a70 con 0x7fabf40b1730 2024-03-25T18:36:29.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2835455886 0 0) 0x7fabe0003520 con 0x7fabf414be90 2024-03-25T18:36:29.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 >> v1:172.21.15.133:6789/0 conn(0x7fabf40b4d60 legacy=0x7fabf413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 >> v1:172.21.15.55:6789/0 conn(0x7fabf40b1730 legacy=0x7fabf40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 10 monclient: _finish_hunting 0 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 1 monclient: found mon.noname-b 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf40b1030 con 0x7fabf414be90 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 10 monclient: _finish_auth 0 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 10 monclient: _check_auth_tickets 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fabec001a10 con 0x7fabf414be90 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabfb56b640 5 monclient: authenticate success, global_id 4592 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabfb56b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabebfff640 10 monclient: dump: 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:29.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.226+0000 7fabfb56b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:29.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.234+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabec001b70 con 0x7fabf414be90 2024-03-25T18:36:29.235 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.234+0000 7fabebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.234+0000 7fabebfff640 1 -- 172.21.15.55:0/2859876465 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fabec001f80 con 0x7fabf414be90 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.234+0000 7fabebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.234+0000 7fabebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.234+0000 7fabebfff640 10 monclient: dump: 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.235+0000 7fabfb56b640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:29.236 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.235+0000 7fabfb56b640 4 set_mon_vals no callback set 2024-03-25T18:36:29.237 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.235+0000 7fabfb56b640 10 monclient: shutdown 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.235+0000 7fabfb56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.235+0000 7fabfb56b640 1 -- 172.21.15.55:0/2859876465 >> v1:172.21.15.55:6790/0 conn(0x7fabf414be90 legacy=0x7fabf414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.235+0000 7fabfb56b640 1 -- 172.21.15.55:0/2859876465 shutdown_connections 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.236+0000 7fabfb56b640 1 -- 172.21.15.55:0/2859876465 >> 172.21.15.55:0/2859876465 conn(0x7fabf40617a0 msgr2=0x7fabf4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.236+0000 7fabfb56b640 1 -- 172.21.15.55:0/2859876465 shutdown_connections 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.236+0000 7fabfb56b640 1 -- 172.21.15.55:0/2859876465 wait complete. 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.236+0000 7fabfb56b640 10 monclient: build_initial_monmap 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.236+0000 7fabfb56b640 10 monclient: monmap: 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:36:29.238 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:36:29.239 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:29.239 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:36:29.239 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:29.239 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:36:29.239 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:36:29.239 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:29.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.237+0000 7fabfb56b640 1 librados: starting msgr at 2024-03-25T18:36:29.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.237+0000 7fabfb56b640 1 librados: starting objecter 2024-03-25T18:36:29.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.237+0000 7fabfb56b640 1 Processor -- start 2024-03-25T18:36:29.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.237+0000 7fabfb56b640 1 -- start start 2024-03-25T18:36:29.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.237+0000 7fabfb56b640 1 librados: setting wanted keys 2024-03-25T18:36:29.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.237+0000 7fabfb56b640 1 librados: calling monclient init 2024-03-25T18:36:29.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.237+0000 7fabfb56b640 10 monclient: init 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 10 monclient(hunting): picked mon.noname-b con 0x7fabf40b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 10 monclient(hunting): picked mon.noname-c con 0x7fabf40b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 10 monclient(hunting): picked mon.noname-a con 0x7fabf414be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf4140de0 con 0x7fabf414be90 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf4141f60 con 0x7fabf40b1730 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf41408c0 con 0x7fabf40b4d60 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:29.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.238+0000 7fabfb56b640 10 monclient(hunting): authenticate will time out at 7793.496241s 2024-03-25T18:36:29.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.247+0000 7fabf8b06640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fabf40b4d60 0x7fabf41dacb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:45456/0 (socket says 172.21.15.55:45456) 2024-03-25T18:36:29.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.247+0000 7fabf8b06640 1 -- 172.21.15.55:0/2568053627 learned_addr learned my addr 172.21.15.55:0/2568053627 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:29.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.251+0000 7fabe9ffb640 10 client.?.objecter ms_handle_connect 0x7fabf40b4d60 2024-03-25T18:36:29.252 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.251+0000 7fabe9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:29.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.252+0000 7fabe9ffb640 10 client.?.objecter ms_handle_connect 0x7fabf414be90 2024-03-25T18:36:29.253 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.252+0000 7fabe9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:29.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.257+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165027899 0 0) 0x7fabf41408c0 con 0x7fabf40b4d60 2024-03-25T18:36:29.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.257+0000 7fabe9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.257+0000 7fabe9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.257+0000 7fabe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.257+0000 7fabe9ffb640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:36:29.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.257+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd8004eb0 con 0x7fabf40b4d60 2024-03-25T18:36:29.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.266+0000 7fabe9ffb640 10 client.?.objecter ms_handle_connect 0x7fabf40b1730 2024-03-25T18:36:29.267 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.266+0000 7fabe9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.277+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 918245436 0 0) 0x7fabd8004eb0 con 0x7fabf40b4d60 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.277+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabf41408c0 con 0x7fabf40b4d60 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.277+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fabe4001a10 con 0x7fabf40b4d60 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.277+0000 7fabe9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.277+0000 7fabe9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.277+0000 7fabe9ffb640 10 monclient(hunting): dump: 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:29.278 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:29.279 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:29.279 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:29.279 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:29.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.277+0000 7fabe9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:29.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.280+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613366266 0 0) 0x7fabf4141f60 con 0x7fabf40b1730 2024-03-25T18:36:29.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.280+0000 7fabe9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.280+0000 7fabe9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.280+0000 7fabe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.280+0000 7fabe9ffb640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:36:29.281 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.280+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd8004eb0 con 0x7fabf40b1730 2024-03-25T18:36:29.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.284+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451102335 0 0) 0x7fabf4140de0 con 0x7fabf414be90 2024-03-25T18:36:29.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.284+0000 7fabe9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:29.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.284+0000 7fabe9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:29.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.284+0000 7fabe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:29.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.284+0000 7fabe9ffb640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:36:29.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.284+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf4141f60 con 0x7fabf414be90 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3290006381 0 0) 0x7fabf41408c0 con 0x7fabf40b4d60 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 >> v1:172.21.15.133:6789/0 conn(0x7fabf40b1730 legacy=0x7fabf4134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 >> v1:172.21.15.55:6789/0 conn(0x7fabf414be90 legacy=0x7fabf41e3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 1 monclient: found mon.c 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf4137760 con 0x7fabf40b4d60 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabe9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:29.293 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabfb56b640 5 monclient: authenticate success, global_id 4595 2024-03-25T18:36:29.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabfb56b640 10 monclient: _renew_subs 2024-03-25T18:36:29.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabfb56b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:29.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.291+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf4140380 con 0x7fabf40b4d60 2024-03-25T18:36:29.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.292+0000 7fabfb56b640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:29.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.292+0000 7fabfb56b640 10 monclient: _renew_subs 2024-03-25T18:36:29.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.292+0000 7fabfb56b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:29.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.292+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf41356a0 con 0x7fabf40b4d60 2024-03-25T18:36:29.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.292+0000 7fabfb56b640 1 librados: init done 2024-03-25T18:36:29.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.293+0000 7fabfb56b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:29.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.293+0000 7fabfb56b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:29.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.293+0000 7fabfb56b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:29.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.293+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf4140a50 con 0x7fabf40b4d60 2024-03-25T18:36:29.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.635+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabe4001cb0 con 0x7fabf40b4d60 2024-03-25T18:36:29.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.635+0000 7fabe9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:29.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.635+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fabe4003a60 con 0x7fabf40b4d60 2024-03-25T18:36:29.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.635+0000 7fabe9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:29.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.635+0000 7fabe9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:29.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.635+0000 7fabe9ffb640 10 monclient: dump: 2024-03-25T18:36:29.637 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:29.637 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:29.637 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:29.638 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:29.638 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:29.638 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:29.638 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:29.638 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:29.638 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:29.638 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:29.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.635+0000 7fabeb7fe640 4 set_mon_vals no callback set 2024-03-25T18:36:29.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.636+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fabe4003c80 con 0x7fabf40b4d60 2024-03-25T18:36:29.639 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.637+0000 7fabf9307640 10 monclient: get_auth_request con 0x7fabd806dea0 auth_method 0 2024-03-25T18:36:29.639 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.637+0000 7fabe9ffb640 10 client.4595.objecter ms_handle_connect 0x7fabd806dea0 2024-03-25T18:36:29.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.644+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fabe4054890 con 0x7fabf40b4d60 2024-03-25T18:36:29.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.644+0000 7fabe9ffb640 10 client.4595.objecter ms_dispatch 0x7fabf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:29.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.644+0000 7fabe9ffb640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:29.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.644+0000 7fabe9ffb640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:29.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.644+0000 7fabe9ffb640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-25T18:36:29.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.644+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabe40503a0 con 0x7fabf40b4d60 2024-03-25T18:36:29.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.644+0000 7fabe9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:29.646 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.644+0000 7fabe9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:29.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.876+0000 7fabfb56b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:29.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.876+0000 7fabfb56b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:29.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.876+0000 7fabfb56b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:36:29.877 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.876+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabf40627b0 con 0x7fabf40b4d60 2024-03-25T18:36:29.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.924+0000 7fabe9ffb640 1 -- 172.21.15.55:0/2568053627 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fabe40522d0 con 0x7fabf40b4d60 2024-03-25T18:36:29.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.924+0000 7fabe9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:29.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.924+0000 7fabe9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:29.925 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:36:29.925 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 10 librados: watch_flush enter 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 10 librados: watch_flush exit 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 10 client.4595.objecter successfully canceled tick 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 >> v1:172.21.15.55:6800/286536064 conn(0x7fabd806dea0 legacy=0x7fabd8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 10 monclient: shutdown 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 >> v1:172.21.15.55:6790/0 conn(0x7fabf40b4d60 legacy=0x7fabf41dacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 shutdown_connections 2024-03-25T18:36:29.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 >> 172.21.15.55:0/2568053627 conn(0x7fabf40617a0 msgr2=0x7fabf413c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:29.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 shutdown_connections 2024-03-25T18:36:29.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 1 -- 172.21.15.55:0/2568053627 wait complete. 2024-03-25T18:36:29.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:29.926+0000 7fabfb56b640 1 librados: shutdown 2024-03-25T18:36:29.940 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:36.383 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:08:46.552 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:36.390 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:08:46.560 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:36.421 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:08:46.591 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:36.443 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:46.580 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:36.454 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:46.592 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:36.456 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:08:46.626 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:36.464 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:08:46.634 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:36.487 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:08:46.625 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:36.502 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:08:46.672 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:36.745 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.867+0000 7f3919e63640 10 monclient: get_monmap_and_config 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.867+0000 7f3919e63640 10 monclient: build_initial_monmap 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.867+0000 7f3919e63640 10 monclient: monmap: 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:36:36.868680+0000 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:36:36.868680+0000 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:36:36.869 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:36:36.870 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.868+0000 7f3919e63640 1 Processor -- start 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.868+0000 7f3919e63640 1 -- start start 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.868+0000 7f3919e63640 10 monclient: init 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 10 monclient(hunting): picked mon.noname-c con 0x7f39140b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 10 monclient(hunting): picked mon.noname-a con 0x7f39140b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 10 monclient(hunting): picked mon.noname-b con 0x7f391414be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39140af940 con 0x7f39140b1730 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39140b0a70 con 0x7f39140b4d60 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39140b0d50 con 0x7f391414be90 2024-03-25T18:36:36.871 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:36.872 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.869+0000 7f3919e63640 10 monclient(hunting): authenticate will time out at 7801.127036s 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.885+0000 7f3913fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f391414be90 0x7f391414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:50464/0 (socket says 172.21.15.55:50464) 2024-03-25T18:36:36.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.885+0000 7f3913fff640 1 -- 172.21.15.55:0/1550411752 learned_addr learned my addr 172.21.15.55:0/1550411752 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:36.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.894+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719625251 0 0) 0x7f39140af940 con 0x7f39140b1730 2024-03-25T18:36:36.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.894+0000 7f39127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:36.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.894+0000 7f39127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:36.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.894+0000 7f39127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:36.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.894+0000 7f39127fc640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:36:36.895 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.894+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38fc003520 con 0x7f39140b1730 2024-03-25T18:36:36.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.896+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300793794 0 0) 0x7f39140b0a70 con 0x7f39140b4d60 2024-03-25T18:36:36.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.896+0000 7f39127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:36.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.896+0000 7f39127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:36.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.896+0000 7f39127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:36.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.896+0000 7f39127fc640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:36:36.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.896+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39140af940 con 0x7f39140b4d60 2024-03-25T18:36:36.901 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 362978785 0 0) 0x7f38fc003520 con 0x7f39140b1730 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 >> v1:172.21.15.55:6790/0 conn(0x7f391414be90 legacy=0x7f391414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 >> v1:172.21.15.133:6789/0 conn(0x7f39140b4d60 legacy=0x7f391413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 1 monclient: found mon.noname-a 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39140b1030 con 0x7f39140b1730 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3908001690 con 0x7f39140b1730 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:36.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f39127fc640 10 monclient: dump: 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f3919e63640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:36:36.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.900+0000 7f3919e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:36.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.911+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39080017f0 con 0x7f39140b1730 2024-03-25T18:36:36.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.911+0000 7f39127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:36.912 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.911+0000 7f39127fc640 1 -- 172.21.15.55:0/1550411752 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3908001c00 con 0x7f39140b1730 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.911+0000 7f39127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.911+0000 7f39127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.911+0000 7f39127fc640 10 monclient: dump: 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:36.913 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:36.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.911+0000 7f3919e63640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:36.914 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.911+0000 7f3919e63640 4 set_mon_vals no callback set 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.912+0000 7f3919e63640 10 monclient: shutdown 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.912+0000 7f3919e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.912+0000 7f3919e63640 1 -- 172.21.15.55:0/1550411752 >> v1:172.21.15.55:6789/0 conn(0x7f39140b1730 legacy=0x7f39140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.912+0000 7f3919e63640 1 -- 172.21.15.55:0/1550411752 shutdown_connections 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.912+0000 7f3919e63640 1 -- 172.21.15.55:0/1550411752 >> 172.21.15.55:0/1550411752 conn(0x7f39140617a0 msgr2=0x7f3914061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.912+0000 7f3919e63640 1 -- 172.21.15.55:0/1550411752 shutdown_connections 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.912+0000 7f3919e63640 1 -- 172.21.15.55:0/1550411752 wait complete. 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.913+0000 7f3919e63640 10 monclient: build_initial_monmap 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.913+0000 7f3919e63640 10 monclient: monmap: 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:36.916 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.913+0000 7f3919e63640 1 librados: starting msgr at 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.913+0000 7f3919e63640 1 librados: starting objecter 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 1 Processor -- start 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 1 -- start start 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 1 librados: setting wanted keys 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 1 librados: calling monclient init 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 10 monclient: init 2024-03-25T18:36:36.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:36.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:36.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 10 monclient(hunting): picked mon.noname-a con 0x7f39140b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:36.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 10 monclient(hunting): picked mon.noname-b con 0x7f39140b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:36.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 10 monclient(hunting): picked mon.noname-c con 0x7f391414be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:36.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3914144d40 con 0x7f39140b1730 2024-03-25T18:36:36.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.914+0000 7f3919e63640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39141dbac0 con 0x7f39140b4d60 2024-03-25T18:36:36.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.915+0000 7f3919e63640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3914135d30 con 0x7f391414be90 2024-03-25T18:36:36.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.915+0000 7f3919e63640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:36.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.915+0000 7f3919e63640 10 monclient(hunting): authenticate will time out at 7801.172392s 2024-03-25T18:36:36.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.920+0000 7f39137fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f39140b1730 0x7f39141db3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:55834/0 (socket says 172.21.15.55:55834) 2024-03-25T18:36:36.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.920+0000 7f39137fe640 1 -- 172.21.15.55:0/1085896376 learned_addr learned my addr 172.21.15.55:0/1085896376 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:36.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.925+0000 7f38fbfff640 10 client.?.objecter ms_handle_connect 0x7f39140b1730 2024-03-25T18:36:36.926 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.925+0000 7f38fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:36.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.931+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047942934 0 0) 0x7f3914144d40 con 0x7f39140b1730 2024-03-25T18:36:36.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.931+0000 7f38fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:36.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.931+0000 7f38fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:36.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.931+0000 7f38fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:36.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.931+0000 7f38fbfff640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:36:36.932 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.931+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3900004eb0 con 0x7f39140b1730 2024-03-25T18:36:36.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.935+0000 7f38fbfff640 10 client.?.objecter ms_handle_connect 0x7f39140b4d60 2024-03-25T18:36:36.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.935+0000 7f38fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:36.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.937+0000 7f38fbfff640 10 client.?.objecter ms_handle_connect 0x7f391414be90 2024-03-25T18:36:36.938 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.937+0000 7f38fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.943+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2713015607 0 0) 0x7f3900004eb0 con 0x7f39140b1730 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.943+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3914144d40 con 0x7f39140b1730 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.943+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3904001a10 con 0x7f39140b1730 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.943+0000 7f38fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.943+0000 7f38fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.943+0000 7f38fbfff640 10 monclient(hunting): dump: 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:36.944 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:36.945 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:36.945 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:36.945 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:36.945 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:36.945 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:36.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.943+0000 7f38fbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:36.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863680645 0 0) 0x7f3914144d40 con 0x7f39140b1730 2024-03-25T18:36:36.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 >> v1:172.21.15.55:6790/0 conn(0x7f391414be90 legacy=0x7f39141e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 >> v1:172.21.15.133:6789/0 conn(0x7f39140b4d60 legacy=0x7f3914144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:36.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:36:36.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 1 monclient: found mon.a 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39141382f0 con 0x7f39140b1730 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 10 monclient: _finish_auth 0 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f38fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f3919e63640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f3919e63640 10 monclient: _renew_subs 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f3919e63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3914145670 con 0x7f39140b1730 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f3919e63640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f3919e63640 10 monclient: _renew_subs 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f3919e63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:36:36.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39141447c0 con 0x7f39140b1730 2024-03-25T18:36:36.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.950+0000 7f3919e63640 1 librados: init done 2024-03-25T18:36:36.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.951+0000 7f3919e63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:36.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.951+0000 7f3919e63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:36.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.951+0000 7f3919e63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:36:36.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:36.951+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39140627b0 con 0x7f39140b1730 2024-03-25T18:36:37.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.275+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3904001c90 con 0x7f39140b1730 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.275+0000 7f38fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.275+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3904003a60 con 0x7f39140b1730 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.275+0000 7f38fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.275+0000 7f38fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.275+0000 7f38fbfff640 10 monclient: dump: 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:37.276 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:37.277 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:37.277 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:37.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.275+0000 7f3911ffb640 4 set_mon_vals no callback set 2024-03-25T18:36:37.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.276+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f3904003c80 con 0x7f39140b1730 2024-03-25T18:36:37.277 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.276+0000 7f3912ffd640 10 monclient: get_auth_request con 0x7f390006dd10 auth_method 0 2024-03-25T18:36:37.278 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.276+0000 7f38fbfff640 10 client.4617.objecter ms_handle_connect 0x7f390006dd10 2024-03-25T18:36:37.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.284+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f3904054890 con 0x7f39140b1730 2024-03-25T18:36:37.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.284+0000 7f38fbfff640 10 client.4617.objecter ms_dispatch 0x7f3914000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:37.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.284+0000 7f38fbfff640 3 client.4617.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:37.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.284+0000 7f38fbfff640 3 client.4617.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:37.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.284+0000 7f38fbfff640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-25T18:36:37.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.285+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39040503a0 con 0x7f39140b1730 2024-03-25T18:36:37.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.285+0000 7f38fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:37.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.285+0000 7f38fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:37.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.534+0000 7f3919e63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:37.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.534+0000 7f3919e63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:37.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.534+0000 7f3919e63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:36:37.535 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.534+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3914069f40 con 0x7f39140b1730 2024-03-25T18:36:37.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.558+0000 7f38fbfff640 1 -- 172.21.15.55:0/1085896376 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f39040522d0 con 0x7f39140b1730 2024-03-25T18:36:37.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.558+0000 7f38fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:37.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.558+0000 7f38fbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:37.559 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:36:37.559 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:37.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 10 librados: watch_flush enter 2024-03-25T18:36:37.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 10 librados: watch_flush exit 2024-03-25T18:36:37.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-25T18:36:37.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 10 client.4617.objecter successfully canceled tick 2024-03-25T18:36:37.562 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 >> v1:172.21.15.55:6800/286536064 conn(0x7f390006dd10 legacy=0x7f39000701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:37.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 10 monclient: shutdown 2024-03-25T18:36:37.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:37.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 >> v1:172.21.15.55:6789/0 conn(0x7f39140b1730 legacy=0x7f39141db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:37.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 shutdown_connections 2024-03-25T18:36:37.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 >> 172.21.15.55:0/1085896376 conn(0x7f39140617a0 msgr2=0x7f39140b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:37.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 shutdown_connections 2024-03-25T18:36:37.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 1 -- 172.21.15.55:0/1085896376 wait complete. 2024-03-25T18:36:37.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:37.561+0000 7f3919e63640 1 librados: shutdown 2024-03-25T18:36:37.573 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:44.380 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:44.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.495+0000 7f4cf177a640 10 monclient: get_monmap_and_config 2024-03-25T18:36:44.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.495+0000 7f4cf177a640 10 monclient: build_initial_monmap 2024-03-25T18:36:44.496 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.495+0000 7f4cf177a640 10 monclient: monmap: 2024-03-25T18:36:44.496 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:44.496 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:44.496 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:36:44.496287+0000 2024-03-25T18:36:44.497 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:36:44.496287+0000 2024-03-25T18:36:44.497 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:44.497 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:44.497 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:44.497 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:36:44.497 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:36:44.497 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:44.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 1 Processor -- start 2024-03-25T18:36:44.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 1 -- start start 2024-03-25T18:36:44.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 10 monclient: init 2024-03-25T18:36:44.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:44.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:44.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 10 monclient(hunting): picked mon.noname-b con 0x7f4cec0a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:44.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 10 monclient(hunting): picked mon.noname-c con 0x7f4cec13f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:44.499 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 10 monclient(hunting): picked mon.noname-a con 0x7f4cec1430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:44.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec142520 con 0x7f4cec1430c0 2024-03-25T18:36:44.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec0b28a0 con 0x7f4cec13f710 2024-03-25T18:36:44.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec0b2a30 con 0x7f4cec0a98f0 2024-03-25T18:36:44.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:44.500 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.496+0000 7f4cf177a640 10 monclient(hunting): authenticate will time out at 7808.754329s 2024-03-25T18:36:44.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.509+0000 7f4cea7fc640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f4cec13f710 0x7f4cec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:52628/0 (socket says 172.21.15.55:52628) 2024-03-25T18:36:44.510 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.509+0000 7f4cea7fc640 1 -- 172.21.15.55:0/1949820402 learned_addr learned my addr 172.21.15.55:0/1949820402 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:44.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.517+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550803203 0 0) 0x7f4cec0b28a0 con 0x7f4cec13f710 2024-03-25T18:36:44.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.517+0000 7f4ce9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:44.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.517+0000 7f4ce9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:44.518 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.517+0000 7f4ce9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:44.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.517+0000 7f4ce9ffb640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:36:44.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.517+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd4003520 con 0x7f4cec13f710 2024-03-25T18:36:44.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.524+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456852834 0 0) 0x7f4cec0b2a30 con 0x7f4cec0a98f0 2024-03-25T18:36:44.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.524+0000 7f4ce9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:44.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.524+0000 7f4ce9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:44.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.524+0000 7f4ce9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:44.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.524+0000 7f4ce9ffb640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:36:44.525 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.524+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cec0b28a0 con 0x7f4cec0a98f0 2024-03-25T18:36:44.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 198442918 0 0) 0x7f4cec0b28a0 con 0x7f4cec0a98f0 2024-03-25T18:36:44.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 >> v1:172.21.15.133:6789/0 conn(0x7f4cec13f710 legacy=0x7f4cec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 >> v1:172.21.15.55:6789/0 conn(0x7f4cec1430c0 legacy=0x7f4cec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:36:44.532 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 1 monclient: found mon.noname-b 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec0b44c0 con 0x7f4cec0a98f0 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4cdc001670 con 0x7f4cec0a98f0 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4ce9ffb640 10 monclient: dump: 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:44.533 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4cf177a640 5 monclient: authenticate success, global_id 4604 2024-03-25T18:36:44.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.531+0000 7f4cf177a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:44.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.551+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cdc0017d0 con 0x7f4cec0a98f0 2024-03-25T18:36:44.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.551+0000 7f4ce9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:44.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.551+0000 7f4ce9ffb640 1 -- 172.21.15.55:0/1949820402 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4cdc001ba0 con 0x7f4cec0a98f0 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.551+0000 7f4cf177a640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.551+0000 7f4cf177a640 4 set_mon_vals no callback set 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.551+0000 7f4ce9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.551+0000 7f4ce9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.551+0000 7f4ce9ffb640 10 monclient: dump: 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:44.553 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:44.554 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:44.554 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:44.554 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.552+0000 7f4cf177a640 10 monclient: shutdown 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.552+0000 7f4cf177a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.552+0000 7f4cf177a640 1 -- 172.21.15.55:0/1949820402 >> v1:172.21.15.55:6790/0 conn(0x7f4cec0a98f0 legacy=0x7f4cec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.552+0000 7f4cf177a640 1 -- 172.21.15.55:0/1949820402 shutdown_connections 2024-03-25T18:36:44.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.552+0000 7f4cf177a640 1 -- 172.21.15.55:0/1949820402 >> 172.21.15.55:0/1949820402 conn(0x7f4cec0b7ea0 msgr2=0x7f4cec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.552+0000 7f4cf177a640 1 -- 172.21.15.55:0/1949820402 shutdown_connections 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.552+0000 7f4cf177a640 1 -- 172.21.15.55:0/1949820402 wait complete. 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.553+0000 7f4cf177a640 10 monclient: build_initial_monmap 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.553+0000 7f4cf177a640 10 monclient: monmap: 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:44.557 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.553+0000 7f4cf177a640 1 librados: starting msgr at 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.553+0000 7f4cf177a640 1 librados: starting objecter 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.553+0000 7f4cf177a640 1 Processor -- start 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.553+0000 7f4cf177a640 1 -- start start 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 1 librados: setting wanted keys 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 1 librados: calling monclient init 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 10 monclient: init 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:44.558 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 10 monclient(hunting): picked mon.noname-a con 0x7f4cec0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:44.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 10 monclient(hunting): picked mon.noname-c con 0x7f4cec13f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:44.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 10 monclient(hunting): picked mon.noname-b con 0x7f4cec1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:44.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec1d7750 con 0x7f4cec0a98f0 2024-03-25T18:36:44.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec136750 con 0x7f4cec1430c0 2024-03-25T18:36:44.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec1d85d0 con 0x7f4cec13f710 2024-03-25T18:36:44.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:44.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.554+0000 7f4cf177a640 10 monclient(hunting): authenticate will time out at 7808.811901s 2024-03-25T18:36:44.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.559+0000 7f4ceb7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f4cec1430c0 0x7f4cec1d4990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:52668/0 (socket says 172.21.15.55:52668) 2024-03-25T18:36:44.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.559+0000 7f4ceb7fe640 1 -- 172.21.15.55:0/1736977650 learned_addr learned my addr 172.21.15.55:0/1736977650 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:44.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.560+0000 7f4cd37fe640 10 client.?.objecter ms_handle_connect 0x7f4cec1430c0 2024-03-25T18:36:44.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.560+0000 7f4cd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:44.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.570+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896476846 0 0) 0x7f4cec136750 con 0x7f4cec1430c0 2024-03-25T18:36:44.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.570+0000 7f4cd37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:44.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.570+0000 7f4cd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:44.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.570+0000 7f4cd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:44.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.570+0000 7f4cd37fe640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:36:44.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.570+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd8004eb0 con 0x7f4cec1430c0 2024-03-25T18:36:44.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.572+0000 7f4cd37fe640 10 client.?.objecter ms_handle_connect 0x7f4cec13f710 2024-03-25T18:36:44.573 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.572+0000 7f4cd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:44.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.576+0000 7f4cd37fe640 10 client.?.objecter ms_handle_connect 0x7f4cec0a98f0 2024-03-25T18:36:44.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.576+0000 7f4cd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:44.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.583+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1505121065 0 0) 0x7f4cd8004eb0 con 0x7f4cec1430c0 2024-03-25T18:36:44.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.583+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cec136750 con 0x7f4cec1430c0 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.583+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4ce4001c90 con 0x7f4cec1430c0 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.583+0000 7f4cd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.583+0000 7f4cd37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.583+0000 7f4cd37fe640 10 monclient(hunting): dump: 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:44.585 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:44.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.583+0000 7f4cd37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.587+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281404470 0 0) 0x7f4cec1d7750 con 0x7f4cec0a98f0 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.587+0000 7f4cd37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.587+0000 7f4cd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.587+0000 7f4cd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.587+0000 7f4cd37fe640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:36:44.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.587+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd8004eb0 con 0x7f4cec0a98f0 2024-03-25T18:36:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.592+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7306826 0 0) 0x7f4cec1d85d0 con 0x7f4cec13f710 2024-03-25T18:36:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.592+0000 7f4cd37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.592+0000 7f4cd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.592+0000 7f4cd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.592+0000 7f4cd37fe640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:36:44.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.592+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cec1d7750 con 0x7f4cec13f710 2024-03-25T18:36:44.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1243682971 0 0) 0x7f4cd8004eb0 con 0x7f4cec0a98f0 2024-03-25T18:36:44.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cec1d85d0 con 0x7f4cec0a98f0 2024-03-25T18:36:44.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4cdc001ae0 con 0x7f4cec0a98f0 2024-03-25T18:36:44.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 10 monclient(hunting): dump: 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:44.605 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 26560497 0 0) 0x7f4cec136750 con 0x7f4cec1430c0 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 >> v1:172.21.15.55:6790/0 conn(0x7f4cec13f710 legacy=0x7f4cec138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 >> v1:172.21.15.55:6789/0 conn(0x7f4cec0a98f0 legacy=0x7f4cec1d7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 1 monclient: found mon.b 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec0ab950 con 0x7f4cec1430c0 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cf177a640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cf177a640 10 monclient: _renew_subs 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cf177a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:44.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec1d7a00 con 0x7f4cec1430c0 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cf177a640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cf177a640 10 monclient: _renew_subs 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cf177a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cec138e60 con 0x7f4cec1430c0 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.603+0000 7f4cf177a640 1 librados: init done 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.604+0000 7f4cf177a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.604+0000 7f4cf177a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:44.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.604+0000 7f4cf177a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:44.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.604+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cec0627b0 con 0x7f4cec1430c0 2024-03-25T18:36:44.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.876+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ce4003990 con 0x7f4cec1430c0 2024-03-25T18:36:44.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.876+0000 7f4cd37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:44.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.876+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4ce4003d20 con 0x7f4cec1430c0 2024-03-25T18:36:44.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.876+0000 7f4cd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:44.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.876+0000 7f4cd37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:44.878 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.876+0000 7f4cd37fe640 10 monclient: dump: 2024-03-25T18:36:44.878 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:44.878 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:44.879 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.879 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:44.879 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:44.879 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:44.879 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:44.879 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:44.879 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:44.879 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:44.879 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.876+0000 7f4ce97fa640 4 set_mon_vals no callback set 2024-03-25T18:36:44.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.877+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f4ce4003f40 con 0x7f4cec1430c0 2024-03-25T18:36:44.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.878+0000 7f4ceaffd640 10 monclient: get_auth_request con 0x7f4cd806e020 auth_method 0 2024-03-25T18:36:44.880 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.878+0000 7f4cd37fe640 10 client.4594.objecter ms_handle_connect 0x7f4cd806e020 2024-03-25T18:36:44.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.883+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f4ce4054a60 con 0x7f4cec1430c0 2024-03-25T18:36:44.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.883+0000 7f4cd37fe640 10 client.4594.objecter ms_dispatch 0x7f4cec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:44.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.883+0000 7f4cd37fe640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:44.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.883+0000 7f4cd37fe640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:44.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.883+0000 7f4cd37fe640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T18:36:44.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.884+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ce4050570 con 0x7f4cec1430c0 2024-03-25T18:36:44.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.884+0000 7f4cd37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:44.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:44.884+0000 7f4cd37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:45.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.106+0000 7f4cf177a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:45.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.106+0000 7f4cf177a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:45.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.106+0000 7f4cf177a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:45.107 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.106+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cec0b7180 con 0x7f4cec1430c0 2024-03-25T18:36:45.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.136+0000 7f4cd37fe640 1 -- 172.21.15.55:0/1736977650 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f4ce40524a0 con 0x7f4cec1430c0 2024-03-25T18:36:45.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.136+0000 7f4cd37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:45.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.136+0000 7f4cd37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:45.137 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:36:45.137 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:45.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.138+0000 7f4cf177a640 10 librados: watch_flush enter 2024-03-25T18:36:45.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.138+0000 7f4cf177a640 10 librados: watch_flush exit 2024-03-25T18:36:45.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.138+0000 7f4cf177a640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T18:36:45.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.138+0000 7f4cf177a640 10 client.4594.objecter successfully canceled tick 2024-03-25T18:36:45.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.138+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 >> v1:172.21.15.55:6800/286536064 conn(0x7f4cd806e020 legacy=0x7f4cd80704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:45.140 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.138+0000 7f4cf177a640 10 monclient: shutdown 2024-03-25T18:36:45.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.138+0000 7f4cf177a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:45.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.138+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 >> v1:172.21.15.133:6789/0 conn(0x7f4cec1430c0 legacy=0x7f4cec1d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:45.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.139+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 shutdown_connections 2024-03-25T18:36:45.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.139+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 >> 172.21.15.55:0/1736977650 conn(0x7f4cec0b7ea0 msgr2=0x7f4cec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:45.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.139+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 shutdown_connections 2024-03-25T18:36:45.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.139+0000 7f4cf177a640 1 -- 172.21.15.55:0/1736977650 wait complete. 2024-03-25T18:36:45.141 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:45.139+0000 7f4cf177a640 1 librados: shutdown 2024-03-25T18:36:45.151 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:51.958 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:52.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.074+0000 7f2a61b1c640 10 monclient: get_monmap_and_config 2024-03-25T18:36:52.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.074+0000 7f2a61b1c640 10 monclient: build_initial_monmap 2024-03-25T18:36:52.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.074+0000 7f2a61b1c640 10 monclient: monmap: 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:36:52.076522+0000 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:36:52.076522+0000 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:36:52.077 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:52.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.075+0000 7f2a61b1c640 1 Processor -- start 2024-03-25T18:36:52.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.075+0000 7f2a61b1c640 1 -- start start 2024-03-25T18:36:52.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.075+0000 7f2a61b1c640 10 monclient: init 2024-03-25T18:36:52.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:52.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:52.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f2a5c0a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:52.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f2a5c13f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:52.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f2a5c1430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:52.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c142520 con 0x7f2a5c13f710 2024-03-25T18:36:52.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c0b28a0 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c0b2a30 con 0x7f2a5c1430c0 2024-03-25T18:36:52.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:52.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.076+0000 7f2a61b1c640 10 monclient(hunting): authenticate will time out at 7816.335249s 2024-03-25T18:36:52.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.085+0000 7f2a5bfff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f2a5c1430c0 0x7f2a5c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:51332/0 (socket says 172.21.15.55:51332) 2024-03-25T18:36:52.086 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.085+0000 7f2a5bfff640 1 -- 172.21.15.55:0/96641022 learned_addr learned my addr 172.21.15.55:0/96641022 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:52.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.106+0000 7f2a5a7fc640 1 -- 172.21.15.55:0/96641022 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976542776 0 0) 0x7f2a5c0b28a0 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.106+0000 7f2a5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:52.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.106+0000 7f2a5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:52.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.106+0000 7f2a5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:52.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.106+0000 7f2a5a7fc640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:36:52.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.106+0000 7f2a5a7fc640 1 -- 172.21.15.55:0/96641022 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a44003520 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 1 -- 172.21.15.55:0/96641022 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3500885588 0 0) 0x7f2a44003520 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 1 -- 172.21.15.55:0/96641022 >> v1:172.21.15.55:6790/0 conn(0x7f2a5c1430c0 legacy=0x7f2a5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 1 -- 172.21.15.55:0/96641022 >> v1:172.21.15.55:6789/0 conn(0x7f2a5c13f710 legacy=0x7f2a5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:52.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:36:52.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:52.116 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:36:52.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 1 -- 172.21.15.55:0/96641022 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c0b44c0 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:52.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:52.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:52.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 1 -- 172.21.15.55:0/96641022 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a4c001690 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:52.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:52.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a5a7fc640 10 monclient: dump: 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a61b1c640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:36:52.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.114+0000 7f2a61b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:52.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a5a7fc640 1 -- 172.21.15.55:0/96641022 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a4c0017f0 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:52.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a5a7fc640 1 -- 172.21.15.55:0/96641022 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a4c001c00 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:52.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:52.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a5a7fc640 10 monclient: dump: 2024-03-25T18:36:52.130 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:52.130 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a61b1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:52.131 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a61b1c640 4 set_mon_vals no callback set 2024-03-25T18:36:52.133 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a61b1c640 10 monclient: shutdown 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a61b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a61b1c640 1 -- 172.21.15.55:0/96641022 >> v1:172.21.15.133:6789/0 conn(0x7f2a5c0a98f0 legacy=0x7f2a5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a61b1c640 1 -- 172.21.15.55:0/96641022 shutdown_connections 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.128+0000 7f2a61b1c640 1 -- 172.21.15.55:0/96641022 >> 172.21.15.55:0/96641022 conn(0x7f2a5c0b7ea0 msgr2=0x7f2a5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.129+0000 7f2a61b1c640 1 -- 172.21.15.55:0/96641022 shutdown_connections 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.129+0000 7f2a61b1c640 1 -- 172.21.15.55:0/96641022 wait complete. 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.129+0000 7f2a61b1c640 10 monclient: build_initial_monmap 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.129+0000 7f2a61b1c640 10 monclient: monmap: 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:52.134 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.129+0000 7f2a61b1c640 1 librados: starting msgr at 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.129+0000 7f2a61b1c640 1 librados: starting objecter 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 1 Processor -- start 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 1 -- start start 2024-03-25T18:36:52.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 1 librados: setting wanted keys 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 1 librados: calling monclient init 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 10 monclient: init 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f2a5c0a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f2a5c13f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f2a5c1430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c137590 con 0x7f2a5c1430c0 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c1db110 con 0x7f2a5c13f710 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c1369c0 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:52.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.130+0000 7f2a61b1c640 10 monclient(hunting): authenticate will time out at 7816.389299s 2024-03-25T18:36:52.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.147+0000 7f2a5bfff640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f2a5c1430c0 0x7f2a5c1e2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:44772/0 (socket says 172.21.15.55:44772) 2024-03-25T18:36:52.149 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.147+0000 7f2a5bfff640 1 -- 172.21.15.55:0/1391123768 learned_addr learned my addr 172.21.15.55:0/1391123768 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:52.156 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.154+0000 7f2a43fff640 10 client.?.objecter ms_handle_connect 0x7f2a5c1430c0 2024-03-25T18:36:52.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.154+0000 7f2a43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:52.166 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.165+0000 7f2a43fff640 10 client.?.objecter ms_handle_connect 0x7f2a5c13f710 2024-03-25T18:36:52.167 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.165+0000 7f2a43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:52.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.167+0000 7f2a43fff640 10 client.?.objecter ms_handle_connect 0x7f2a5c0a98f0 2024-03-25T18:36:52.169 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.167+0000 7f2a43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:52.170 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.169+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757124438 0 0) 0x7f2a5c1db110 con 0x7f2a5c13f710 2024-03-25T18:36:52.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.169+0000 7f2a43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:52.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.169+0000 7f2a43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:52.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.169+0000 7f2a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:52.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.169+0000 7f2a43fff640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:36:52.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.169+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a48004eb0 con 0x7f2a5c13f710 2024-03-25T18:36:52.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.179+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445804677 0 0) 0x7f2a5c137590 con 0x7f2a5c1430c0 2024-03-25T18:36:52.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.179+0000 7f2a43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:52.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.179+0000 7f2a43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:52.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.179+0000 7f2a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:52.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.179+0000 7f2a43fff640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:36:52.181 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.179+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a5c1db110 con 0x7f2a5c1430c0 2024-03-25T18:36:52.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.183+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3784733309 0 0) 0x7f2a48004eb0 con 0x7f2a5c13f710 2024-03-25T18:36:52.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.183+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a5c137590 con 0x7f2a5c13f710 2024-03-25T18:36:52.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.183+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a50001930 con 0x7f2a5c13f710 2024-03-25T18:36:52.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.183+0000 7f2a43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:52.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.183+0000 7f2a43fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:52.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.183+0000 7f2a43fff640 10 monclient(hunting): dump: 2024-03-25T18:36:52.185 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:52.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.183+0000 7f2a43fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:52.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.185+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480444364 0 0) 0x7f2a5c1369c0 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.185+0000 7f2a43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:52.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.185+0000 7f2a43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:52.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.185+0000 7f2a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:52.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.185+0000 7f2a43fff640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:36:52.188 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.185+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a48004eb0 con 0x7f2a5c0a98f0 2024-03-25T18:36:52.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1521232404 0 0) 0x7f2a5c137590 con 0x7f2a5c13f710 2024-03-25T18:36:52.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 >> v1:172.21.15.55:6790/0 conn(0x7f2a5c0a98f0 legacy=0x7f2a5c1daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 >> v1:172.21.15.55:6789/0 conn(0x7f2a5c1430c0 legacy=0x7f2a5c1e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 10 monclient: _finish_hunting 0 2024-03-25T18:36:52.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 1 monclient: found mon.b 2024-03-25T18:36:52.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:52.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:52.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c136b50 con 0x7f2a5c13f710 2024-03-25T18:36:52.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 10 monclient: _finish_auth 0 2024-03-25T18:36:52.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 10 monclient: _check_auth_tickets 2024-03-25T18:36:52.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:52.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a61b1c640 5 monclient: authenticate success, global_id 4600 2024-03-25T18:36:52.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a61b1c640 10 monclient: _renew_subs 2024-03-25T18:36:52.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a61b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:52.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a5c137a70 con 0x7f2a5c13f710 2024-03-25T18:36:52.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a61b1c640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:52.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a61b1c640 10 monclient: _renew_subs 2024-03-25T18:36:52.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a61b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:52.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a5c13b970 con 0x7f2a5c13f710 2024-03-25T18:36:52.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.195+0000 7f2a61b1c640 1 librados: init done 2024-03-25T18:36:52.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.196+0000 7f2a61b1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:52.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.196+0000 7f2a61b1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:52.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.196+0000 7f2a61b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:52.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.196+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a5c0627b0 con 0x7f2a5c13f710 2024-03-25T18:36:52.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.486+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a50003610 con 0x7f2a5c13f710 2024-03-25T18:36:52.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.486+0000 7f2a43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:52.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.486+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2a500039a0 con 0x7f2a5c13f710 2024-03-25T18:36:52.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.486+0000 7f2a43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:52.488 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.486+0000 7f2a43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.486+0000 7f2a43fff640 10 monclient: dump: 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:52.489 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.486+0000 7f2a59ffb640 4 set_mon_vals no callback set 2024-03-25T18:36:52.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.487+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f2a50003bc0 con 0x7f2a5c13f710 2024-03-25T18:36:52.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.487+0000 7f2a5b7fe640 10 monclient: get_auth_request con 0x7f2a4806de00 auth_method 0 2024-03-25T18:36:52.490 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.488+0000 7f2a43fff640 10 client.4600.objecter ms_handle_connect 0x7f2a4806de00 2024-03-25T18:36:52.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.491+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f2a50053620 con 0x7f2a5c13f710 2024-03-25T18:36:52.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.491+0000 7f2a43fff640 10 client.4600.objecter ms_dispatch 0x7f2a5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:52.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.491+0000 7f2a43fff640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:52.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.491+0000 7f2a43fff640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:52.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.491+0000 7f2a43fff640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T18:36:52.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.493+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a5004f130 con 0x7f2a5c13f710 2024-03-25T18:36:52.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.493+0000 7f2a43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:52.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.493+0000 7f2a43fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:52.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.724+0000 7f2a61b1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:52.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.724+0000 7f2a61b1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:52.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.724+0000 7f2a61b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:52.727 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.724+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a5c069f40 con 0x7f2a5c13f710 2024-03-25T18:36:52.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.745+0000 7f2a43fff640 1 -- 172.21.15.55:0/1391123768 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f2a50051060 con 0x7f2a5c13f710 2024-03-25T18:36:52.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.745+0000 7f2a43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:52.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.745+0000 7f2a43fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:52.747 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:36:52.748 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 10 librados: watch_flush enter 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 10 librados: watch_flush exit 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 10 client.4600.objecter successfully canceled tick 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 >> v1:172.21.15.55:6800/286536064 conn(0x7f2a4806de00 legacy=0x7f2a480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 10 monclient: shutdown 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 >> v1:172.21.15.133:6789/0 conn(0x7f2a5c13f710 legacy=0x7f2a5c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 shutdown_connections 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 >> 172.21.15.55:0/1391123768 conn(0x7f2a5c0b7ea0 msgr2=0x7f2a5c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:52.750 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 shutdown_connections 2024-03-25T18:36:52.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 1 -- 172.21.15.55:0/1391123768 wait complete. 2024-03-25T18:36:52.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:52.747+0000 7f2a61b1c640 1 librados: shutdown 2024-03-25T18:36:52.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:56.451 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:06.588 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:56.453 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:06.623 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:56.454 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:06.592 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:56.459 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:06.630 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:56.489 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:06.659 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:56.502 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:06.639 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:56.522 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:09:06.692 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:56.530 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:09:06.700 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:56.560 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:09:06.730 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:59.569 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:59.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.674+0000 7f292620d640 10 monclient: get_monmap_and_config 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.674+0000 7f292620d640 10 monclient: build_initial_monmap 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.675+0000 7f292620d640 10 monclient: monmap: 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:36:59.676840+0000 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:36:59.676840+0000 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:36:59.677 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.675+0000 7f292620d640 1 Processor -- start 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.675+0000 7f292620d640 1 -- start start 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.675+0000 7f292620d640 10 monclient: init 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 10 monclient(hunting): picked mon.noname-c con 0x7f29200b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 10 monclient(hunting): picked mon.noname-b con 0x7f29200b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 10 monclient(hunting): picked mon.noname-a con 0x7f292014be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29200af940 con 0x7f292014be90 2024-03-25T18:36:59.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29200b0a70 con 0x7f29200b4d60 2024-03-25T18:36:59.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29200b0d50 con 0x7f29200b1730 2024-03-25T18:36:59.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:59.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.676+0000 7f292620d640 10 monclient(hunting): authenticate will time out at 7823.934768s 2024-03-25T18:36:59.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.677+0000 7f291effd640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f29200b1730 0x7f29200b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:47906/0 (socket says 172.21.15.55:47906) 2024-03-25T18:36:59.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.677+0000 7f291effd640 1 -- 172.21.15.55:0/2512999634 learned_addr learned my addr 172.21.15.55:0/2512999634 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:59.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.763+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119517817 0 0) 0x7f29200af940 con 0x7f292014be90 2024-03-25T18:36:59.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.763+0000 7f291e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.763+0000 7f291e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.763+0000 7f291e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.763+0000 7f291e7fc640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:36:59.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.763+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f290c003520 con 0x7f292014be90 2024-03-25T18:36:59.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.782+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974889031 0 0) 0x7f29200b0a70 con 0x7f29200b4d60 2024-03-25T18:36:59.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.782+0000 7f291e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.782+0000 7f291e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.782+0000 7f291e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.782+0000 7f291e7fc640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:36:59.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.782+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29200af940 con 0x7f29200b4d60 2024-03-25T18:36:59.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.796+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326443807 0 0) 0x7f29200b0d50 con 0x7f29200b1730 2024-03-25T18:36:59.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.796+0000 7f291e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.796+0000 7f291e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.796+0000 7f291e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.796+0000 7f291e7fc640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:36:59.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.796+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29200b0a70 con 0x7f29200b1730 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3647674947 0 0) 0x7f290c003520 con 0x7f292014be90 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 >> v1:172.21.15.55:6790/0 conn(0x7f29200b1730 legacy=0x7f29200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 >> v1:172.21.15.133:6789/0 conn(0x7f29200b4d60 legacy=0x7f292013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29200b1030 con 0x7f292014be90 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:59.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2918001a10 con 0x7f292014be90 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f291e7fc640 10 monclient: dump: 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:59.811 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:59.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f292620d640 5 monclient: authenticate success, global_id 4632 2024-03-25T18:36:59.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.808+0000 7f292620d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.899+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2918001b70 con 0x7f292014be90 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f291e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f291e7fc640 1 -- 172.21.15.55:0/2512999634 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2918001f80 con 0x7f292014be90 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f291e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f291e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f291e7fc640 10 monclient: dump: 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:59.902 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:59.903 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:59.903 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:59.903 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:59.903 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:59.903 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:59.903 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:59.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f292620d640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:59.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f292620d640 4 set_mon_vals no callback set 2024-03-25T18:36:59.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f292620d640 10 monclient: shutdown 2024-03-25T18:36:59.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f292620d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:59.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f292620d640 1 -- 172.21.15.55:0/2512999634 >> v1:172.21.15.55:6789/0 conn(0x7f292014be90 legacy=0x7f292014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f292620d640 1 -- 172.21.15.55:0/2512999634 shutdown_connections 2024-03-25T18:36:59.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f292620d640 1 -- 172.21.15.55:0/2512999634 >> 172.21.15.55:0/2512999634 conn(0x7f29200617a0 msgr2=0x7f2920061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:59.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f292620d640 1 -- 172.21.15.55:0/2512999634 shutdown_connections 2024-03-25T18:36:59.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.900+0000 7f292620d640 1 -- 172.21.15.55:0/2512999634 wait complete. 2024-03-25T18:36:59.905 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.901+0000 7f292620d640 10 monclient: build_initial_monmap 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.901+0000 7f292620d640 10 monclient: monmap: 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:59.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.901+0000 7f292620d640 1 librados: starting msgr at 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.901+0000 7f292620d640 1 librados: starting objecter 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.902+0000 7f292620d640 1 Processor -- start 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.902+0000 7f292620d640 1 -- start start 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.902+0000 7f292620d640 1 librados: setting wanted keys 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.902+0000 7f292620d640 1 librados: calling monclient init 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.902+0000 7f292620d640 10 monclient: init 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.902+0000 7f292620d640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.902+0000 7f292620d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.902+0000 7f292620d640 10 monclient(hunting): picked mon.noname-b con 0x7f29200b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.903+0000 7f292620d640 10 monclient(hunting): picked mon.noname-a con 0x7f29200b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.903+0000 7f292620d640 10 monclient(hunting): picked mon.noname-c con 0x7f292014be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.903+0000 7f292620d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2920138100 con 0x7f29200b4d60 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.903+0000 7f292620d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29201383e0 con 0x7f29200b1730 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.903+0000 7f292620d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2920136db0 con 0x7f292014be90 2024-03-25T18:36:59.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.903+0000 7f292620d640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:59.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.903+0000 7f292620d640 10 monclient(hunting): authenticate will time out at 7824.161578s 2024-03-25T18:36:59.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.915+0000 7f291effd640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f29200b4d60 0x7f2920143900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:49886/0 (socket says 172.21.15.55:49886) 2024-03-25T18:36:59.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.915+0000 7f291effd640 1 -- 172.21.15.55:0/1793864650 learned_addr learned my addr 172.21.15.55:0/1793864650 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:36:59.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.916+0000 7f2917fff640 10 client.?.objecter ms_handle_connect 0x7f29200b4d60 2024-03-25T18:36:59.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.916+0000 7f2917fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:59.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.917+0000 7f2917fff640 10 client.?.objecter ms_handle_connect 0x7f29200b1730 2024-03-25T18:36:59.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.917+0000 7f2917fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:59.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.920+0000 7f2917fff640 10 client.?.objecter ms_handle_connect 0x7f292014be90 2024-03-25T18:36:59.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.920+0000 7f2917fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:59.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.926+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262229060 0 0) 0x7f29201383e0 con 0x7f29200b1730 2024-03-25T18:36:59.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.927+0000 7f2917fff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.927+0000 7f2917fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.927+0000 7f2917fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.927+0000 7f2917fff640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:36:59.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.927+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2900004eb0 con 0x7f29200b1730 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.937+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 998331074 0 0) 0x7f2900004eb0 con 0x7f29200b1730 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.938+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29201383e0 con 0x7f29200b1730 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.938+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2908001970 con 0x7f29200b1730 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.938+0000 7f2917fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.938+0000 7f2917fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.938+0000 7f2917fff640 10 monclient(hunting): dump: 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:59.940 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:36:59.941 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:36:59.941 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:36:59.941 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:36:59.941 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:36:59.941 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.938+0000 7f2917fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:59.944 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.943+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670287402 0 0) 0x7f2920136db0 con 0x7f292014be90 2024-03-25T18:36:59.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.943+0000 7f2917fff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.943+0000 7f2917fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.943+0000 7f2917fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.943+0000 7f2917fff640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:36:59.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.943+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2900004eb0 con 0x7f292014be90 2024-03-25T18:36:59.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.945+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381284532 0 0) 0x7f2920138100 con 0x7f29200b4d60 2024-03-25T18:36:59.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.945+0000 7f2917fff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:59.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.945+0000 7f2917fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:59.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.945+0000 7f2917fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:59.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.945+0000 7f2917fff640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:36:59.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.945+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2920136db0 con 0x7f29200b4d60 2024-03-25T18:36:59.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1977617177 0 0) 0x7f29201383e0 con 0x7f29200b1730 2024-03-25T18:36:59.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 >> v1:172.21.15.55:6790/0 conn(0x7f292014be90 legacy=0x7f29201e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 >> v1:172.21.15.55:6789/0 conn(0x7f29200b4d60 legacy=0x7f2920143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 10 monclient: _finish_hunting 0 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 1 monclient: found mon.b 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2920135ff0 con 0x7f29200b1730 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 10 monclient: _finish_auth 0 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 10 monclient: _check_auth_tickets 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f2917fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f292620d640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:36:59.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f292620d640 10 monclient: _renew_subs 2024-03-25T18:36:59.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f292620d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:59.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2920136240 con 0x7f29200b1730 2024-03-25T18:36:59.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f292620d640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:59.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f292620d640 10 monclient: _renew_subs 2024-03-25T18:36:59.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f292620d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:59.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2920136900 con 0x7f29200b1730 2024-03-25T18:36:59.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.947+0000 7f292620d640 1 librados: init done 2024-03-25T18:36:59.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.948+0000 7f292620d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:59.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.948+0000 7f292620d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:59.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.948+0000 7f292620d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:36:59.952 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:36:59.948+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2920136f40 con 0x7f29200b1730 2024-03-25T18:37:00.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.244+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2908003610 con 0x7f29200b1730 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.244+0000 7f2917fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.244+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f2908003960 con 0x7f29200b1730 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.244+0000 7f2917fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.244+0000 7f2917fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.244+0000 7f2917fff640 10 monclient: dump: 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:00.246 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:00.247 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:00.247 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:00.247 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:00.247 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:00.247 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:00.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.244+0000 7f291dffb640 4 set_mon_vals no callback set 2024-03-25T18:37:00.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.245+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f29080199b0 con 0x7f29200b1730 2024-03-25T18:37:00.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.246+0000 7f291effd640 10 monclient: get_auth_request con 0x7f290006df10 auth_method 0 2024-03-25T18:37:00.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.246+0000 7f2917fff640 10 client.4606.objecter ms_handle_connect 0x7f290006df10 2024-03-25T18:37:00.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.250+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f29080536d0 con 0x7f29200b1730 2024-03-25T18:37:00.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.250+0000 7f2917fff640 10 client.4606.objecter ms_dispatch 0x7f2920000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:37:00.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.250+0000 7f2917fff640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:37:00.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.250+0000 7f2917fff640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:37:00.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.250+0000 7f2917fff640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T18:37:00.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.250+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f290804f010 con 0x7f29200b1730 2024-03-25T18:37:00.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.250+0000 7f2917fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:00.251 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.250+0000 7f2917fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:00.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.509+0000 7f292620d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:00.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.509+0000 7f292620d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:00.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.509+0000 7f292620d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:00.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.509+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29200b2270 con 0x7f29200b1730 2024-03-25T18:37:00.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.545+0000 7f2917fff640 1 -- 172.21.15.55:0/1793864650 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f2908051110 con 0x7f29200b1730 2024-03-25T18:37:00.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.545+0000 7f2917fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:00.546 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.545+0000 7f2917fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:00.547 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:37:00.547 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:00.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 10 librados: watch_flush enter 2024-03-25T18:37:00.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 10 librados: watch_flush exit 2024-03-25T18:37:00.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T18:37:00.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 10 client.4606.objecter successfully canceled tick 2024-03-25T18:37:00.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 >> v1:172.21.15.55:6800/286536064 conn(0x7f290006df10 legacy=0x7f29000703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:00.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 10 monclient: shutdown 2024-03-25T18:37:00.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:00.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 >> v1:172.21.15.133:6789/0 conn(0x7f29200b1730 legacy=0x7f29201427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:00.551 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 shutdown_connections 2024-03-25T18:37:00.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 >> 172.21.15.55:0/1793864650 conn(0x7f29200617a0 msgr2=0x7f292013c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:00.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 shutdown_connections 2024-03-25T18:37:00.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 1 -- 172.21.15.55:0/1793864650 wait complete. 2024-03-25T18:37:00.552 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:00.547+0000 7f292620d640 1 librados: shutdown 2024-03-25T18:37:00.560 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:07.366 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:07.482 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.481+0000 7fa4e06f6640 10 monclient: get_monmap_and_config 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.481+0000 7fa4e06f6640 10 monclient: build_initial_monmap 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.481+0000 7fa4e06f6640 10 monclient: monmap: 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:37:07.482610+0000 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:37:07.482610+0000 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:37:07.483 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.482+0000 7fa4e06f6640 1 Processor -- start 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.482+0000 7fa4e06f6640 1 -- start start 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.482+0000 7fa4e06f6640 10 monclient: init 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa4d80a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa4d813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa4d81430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d8142520 con 0x7fa4d80a98f0 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d80b28a0 con 0x7fa4d813f710 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d80b2a30 con 0x7fa4d81430c0 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:07.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.483+0000 7fa4e06f6640 10 monclient(hunting): authenticate will time out at 7831.741060s 2024-03-25T18:37:07.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.494+0000 7fa4de492640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa4d80a98f0 0x7fa4d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54654/0 (socket says 172.21.15.55:54654) 2024-03-25T18:37:07.495 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.494+0000 7fa4de492640 1 -- 172.21.15.55:0/2697145654 learned_addr learned my addr 172.21.15.55:0/2697145654 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:07.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.510+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505782891 0 0) 0x7fa4d8142520 con 0x7fa4d80a98f0 2024-03-25T18:37:07.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.510+0000 7fa4dd490640 10 monclient(hunting): _negotiate 2024-03-25T18:37:07.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.510+0000 7fa4dd490640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:07.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.510+0000 7fa4dd490640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:07.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.510+0000 7fa4dd490640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:37:07.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.510+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c4003520 con 0x7fa4d80a98f0 2024-03-25T18:37:07.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.512+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621456512 0 0) 0x7fa4d80b2a30 con 0x7fa4d81430c0 2024-03-25T18:37:07.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.512+0000 7fa4dd490640 10 monclient(hunting): _negotiate 2024-03-25T18:37:07.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.512+0000 7fa4dd490640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:07.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.512+0000 7fa4dd490640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:07.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.512+0000 7fa4dd490640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:37:07.513 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.512+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d8142520 con 0x7fa4d81430c0 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 878055172 0 0) 0x7fa4d8142520 con 0x7fa4d81430c0 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 >> v1:172.21.15.133:6789/0 conn(0x7fa4d813f710 legacy=0x7fa4d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 >> v1:172.21.15.55:6789/0 conn(0x7fa4d80a98f0 legacy=0x7fa4d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 10 monclient: _finish_hunting 0 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 1 monclient: found mon.noname-b 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d80b44c0 con 0x7fa4d81430c0 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 10 monclient: _finish_auth 0 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 10 monclient: _check_auth_tickets 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fa4d00019f0 con 0x7fa4d81430c0 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4e06f6640 5 monclient: authenticate success, global_id 4622 2024-03-25T18:37:07.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4e06f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4dd490640 10 monclient: dump: 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:07.520 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.517+0000 7fa4e06f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.521+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4d0001b50 con 0x7fa4d81430c0 2024-03-25T18:37:07.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.521+0000 7fa4dd490640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4e06f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.521+0000 7fa4dd490640 1 -- 172.21.15.55:0/2697145654 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fa4d0001f20 con 0x7fa4d81430c0 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4e06f6640 4 set_mon_vals no callback set 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4dd490640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4dd490640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4dd490640 10 monclient: dump: 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:07.523 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:07.524 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:07.524 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:07.524 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:07.524 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4e06f6640 10 monclient: shutdown 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4e06f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4e06f6640 1 -- 172.21.15.55:0/2697145654 >> v1:172.21.15.55:6790/0 conn(0x7fa4d81430c0 legacy=0x7fa4d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4e06f6640 1 -- 172.21.15.55:0/2697145654 shutdown_connections 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4e06f6640 1 -- 172.21.15.55:0/2697145654 >> 172.21.15.55:0/2697145654 conn(0x7fa4d80b7ea0 msgr2=0x7fa4d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4e06f6640 1 -- 172.21.15.55:0/2697145654 shutdown_connections 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.522+0000 7fa4e06f6640 1 -- 172.21.15.55:0/2697145654 wait complete. 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.523+0000 7fa4e06f6640 10 monclient: build_initial_monmap 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.523+0000 7fa4e06f6640 10 monclient: monmap: 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:07.526 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.523+0000 7fa4e06f6640 1 librados: starting msgr at 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.523+0000 7fa4e06f6640 1 librados: starting objecter 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.523+0000 7fa4e06f6640 1 Processor -- start 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 1 -- start start 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 1 librados: setting wanted keys 2024-03-25T18:37:07.527 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 1 librados: calling monclient init 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 10 monclient: init 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa4d80a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa4d813f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa4d81430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d813aae0 con 0x7fa4d813f710 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d813ad50 con 0x7fa4d81430c0 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d813b030 con 0x7fa4d80a98f0 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:07.528 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.524+0000 7fa4e06f6640 10 monclient(hunting): authenticate will time out at 7831.782095s 2024-03-25T18:37:07.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.529+0000 7fa4ddc91640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fa4d813f710 0x7fa4d81df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54678/0 (socket says 172.21.15.55:54678) 2024-03-25T18:37:07.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.529+0000 7fa4ddc91640 1 -- 172.21.15.55:0/1306456806 learned_addr learned my addr 172.21.15.55:0/1306456806 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:07.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.533+0000 7fa4ceffd640 10 client.?.objecter ms_handle_connect 0x7fa4d813f710 2024-03-25T18:37:07.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.533+0000 7fa4ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:07.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.547+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193201611 0 0) 0x7fa4d813aae0 con 0x7fa4d813f710 2024-03-25T18:37:07.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.547+0000 7fa4ceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:37:07.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.547+0000 7fa4ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:07.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.547+0000 7fa4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:07.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.547+0000 7fa4ceffd640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:37:07.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.547+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4bc004eb0 con 0x7fa4d813f710 2024-03-25T18:37:07.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.549+0000 7fa4ceffd640 10 client.?.objecter ms_handle_connect 0x7fa4d80a98f0 2024-03-25T18:37:07.550 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.549+0000 7fa4ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:07.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.555+0000 7fa4ceffd640 10 client.?.objecter ms_handle_connect 0x7fa4d81430c0 2024-03-25T18:37:07.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.555+0000 7fa4ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:07.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.560+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085929731 0 0) 0x7fa4d813ad50 con 0x7fa4d81430c0 2024-03-25T18:37:07.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.560+0000 7fa4ceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:37:07.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.560+0000 7fa4ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:07.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.560+0000 7fa4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:07.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.560+0000 7fa4ceffd640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:37:07.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.560+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d813aae0 con 0x7fa4d81430c0 2024-03-25T18:37:07.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.564+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719490938 0 0) 0x7fa4d813b030 con 0x7fa4d80a98f0 2024-03-25T18:37:07.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.564+0000 7fa4ceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:37:07.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.564+0000 7fa4ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:07.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.564+0000 7fa4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:07.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.564+0000 7fa4ceffd640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:37:07.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.564+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d813ad50 con 0x7fa4d80a98f0 2024-03-25T18:37:07.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3501418013 0 0) 0x7fa4bc004eb0 con 0x7fa4d813f710 2024-03-25T18:37:07.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4d813b030 con 0x7fa4d813f710 2024-03-25T18:37:07.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fa4c8001a30 con 0x7fa4d813f710 2024-03-25T18:37:07.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:07.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:37:07.581 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 10 monclient(hunting): dump: 2024-03-25T18:37:07.581 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:07.581 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2870963720 0 0) 0x7fa4d813aae0 con 0x7fa4d81430c0 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4bc004eb0 con 0x7fa4d81430c0 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fa4d0001e60 con 0x7fa4d81430c0 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:07.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.579+0000 7fa4ceffd640 10 monclient(hunting): dump: 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:07.583 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:07.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2289295606 0 0) 0x7fa4bc004eb0 con 0x7fa4d81430c0 2024-03-25T18:37:07.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 >> v1:172.21.15.55:6790/0 conn(0x7fa4d80a98f0 legacy=0x7fa4d813ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:07.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 >> v1:172.21.15.55:6789/0 conn(0x7fa4d813f710 legacy=0x7fa4d81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:07.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:37:07.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 1 monclient: found mon.b 2024-03-25T18:37:07.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:07.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d813b310 con 0x7fa4d81430c0 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 10 monclient: _finish_auth 0 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4e06f6640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4e06f6640 10 monclient: _renew_subs 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4e06f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d813b5f0 con 0x7fa4d81430c0 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4e06f6640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4e06f6640 10 monclient: _renew_subs 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4e06f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d812f6c0 con 0x7fa4d81430c0 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.586+0000 7fa4e06f6640 1 librados: init done 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.587+0000 7fa4e06f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:07.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.587+0000 7fa4e06f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:07.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.587+0000 7fa4e06f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:07.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.587+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d80627b0 con 0x7fa4d81430c0 2024-03-25T18:37:07.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.886+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4d0002020 con 0x7fa4d81430c0 2024-03-25T18:37:07.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.886+0000 7fa4ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:07.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.886+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fa4d0003dc0 con 0x7fa4d81430c0 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.886+0000 7fa4ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.886+0000 7fa4ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.886+0000 7fa4ceffd640 10 monclient: dump: 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:07.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.886+0000 7fa4dcc8f640 4 set_mon_vals no callback set 2024-03-25T18:37:07.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.887+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fa4d0019d90 con 0x7fa4d81430c0 2024-03-25T18:37:07.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.888+0000 7fa4de492640 10 monclient: get_auth_request con 0x7fa4bc06df70 auth_method 0 2024-03-25T18:37:07.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.888+0000 7fa4ceffd640 10 client.4612.objecter ms_handle_connect 0x7fa4bc06df70 2024-03-25T18:37:07.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.892+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fa4d0053920 con 0x7fa4d81430c0 2024-03-25T18:37:07.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.892+0000 7fa4ceffd640 10 client.4612.objecter ms_dispatch 0x7fa4d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:37:07.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.892+0000 7fa4ceffd640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:37:07.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.892+0000 7fa4ceffd640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:37:07.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.892+0000 7fa4ceffd640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T18:37:07.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.892+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4d004f260 con 0x7fa4d81430c0 2024-03-25T18:37:07.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.892+0000 7fa4ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:07.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:07.892+0000 7fa4ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:08.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.153+0000 7fa4e06f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:08.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.153+0000 7fa4e06f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:08.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.153+0000 7fa4e06f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:08.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.153+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4d813de20 con 0x7fa4d81430c0 2024-03-25T18:37:08.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.179+0000 7fa4ceffd640 1 -- 172.21.15.55:0/1306456806 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fa4d0051360 con 0x7fa4d81430c0 2024-03-25T18:37:08.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.179+0000 7fa4ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:08.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.179+0000 7fa4ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:08.181 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:37:08.181 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:08.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.181+0000 7fa4e06f6640 10 librados: watch_flush enter 2024-03-25T18:37:08.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.181+0000 7fa4e06f6640 10 librados: watch_flush exit 2024-03-25T18:37:08.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.181+0000 7fa4e06f6640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T18:37:08.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.181+0000 7fa4e06f6640 10 client.4612.objecter successfully canceled tick 2024-03-25T18:37:08.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.182+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 >> v1:172.21.15.55:6800/286536064 conn(0x7fa4bc06df70 legacy=0x7fa4bc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:08.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.182+0000 7fa4e06f6640 10 monclient: shutdown 2024-03-25T18:37:08.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.182+0000 7fa4e06f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:08.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.182+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 >> v1:172.21.15.133:6789/0 conn(0x7fa4d81430c0 legacy=0x7fa4d81e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:08.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.182+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 shutdown_connections 2024-03-25T18:37:08.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.182+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 >> 172.21.15.55:0/1306456806 conn(0x7fa4d80b7ea0 msgr2=0x7fa4d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:08.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.182+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 shutdown_connections 2024-03-25T18:37:08.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.182+0000 7fa4e06f6640 1 -- 172.21.15.55:0/1306456806 wait complete. 2024-03-25T18:37:08.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:08.182+0000 7fa4e06f6640 1 librados: shutdown 2024-03-25T18:37:08.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:15.000 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:15.118 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.117+0000 7f9b8000e640 10 monclient: get_monmap_and_config 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.117+0000 7f9b8000e640 10 monclient: build_initial_monmap 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.117+0000 7f9b8000e640 10 monclient: monmap: 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:37:15.118782+0000 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:37:15.118782+0000 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:37:15.119 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:15.120 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.118+0000 7f9b8000e640 1 Processor -- start 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.118+0000 7f9b8000e640 1 -- start start 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.118+0000 7f9b8000e640 10 monclient: init 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 10 monclient(hunting): picked mon.noname-b con 0x7f9b780a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 10 monclient(hunting): picked mon.noname-a con 0x7f9b781430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78142520 con 0x7f9b781430c0 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780b28a0 con 0x7f9b7813f710 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780b2a30 con 0x7f9b780a98f0 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:15.121 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.119+0000 7f9b8000e640 10 monclient(hunting): authenticate will time out at 7839.377101s 2024-03-25T18:37:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.131+0000 7f9b7ddaa640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f9b780a98f0 0x7f9b780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:36254/0 (socket says 172.21.15.55:36254) 2024-03-25T18:37:15.132 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.131+0000 7f9b7ddaa640 1 -- 172.21.15.55:0/1994982530 learned_addr learned my addr 172.21.15.55:0/1994982530 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:15.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.143+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677247951 0 0) 0x7f9b780b2a30 con 0x7f9b780a98f0 2024-03-25T18:37:15.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.143+0000 7f9b7cda8640 10 monclient(hunting): _negotiate 2024-03-25T18:37:15.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.143+0000 7f9b7cda8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:15.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.143+0000 7f9b7cda8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:15.144 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.143+0000 7f9b7cda8640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:37:15.145 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.143+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b64003520 con 0x7f9b780a98f0 2024-03-25T18:37:15.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.147+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860395898 0 0) 0x7f9b780b28a0 con 0x7f9b7813f710 2024-03-25T18:37:15.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.147+0000 7f9b7cda8640 10 monclient(hunting): _negotiate 2024-03-25T18:37:15.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.147+0000 7f9b7cda8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:15.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.147+0000 7f9b7cda8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:15.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.147+0000 7f9b7cda8640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:37:15.148 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.147+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b780b2a30 con 0x7f9b7813f710 2024-03-25T18:37:15.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1499577446 0 0) 0x7f9b64003520 con 0x7f9b780a98f0 2024-03-25T18:37:15.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 >> v1:172.21.15.133:6789/0 conn(0x7f9b7813f710 legacy=0x7f9b781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 >> v1:172.21.15.55:6789/0 conn(0x7f9b781430c0 legacy=0x7f9b7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 10 monclient: _finish_hunting 0 2024-03-25T18:37:15.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 1 monclient: found mon.noname-b 2024-03-25T18:37:15.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:15.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:37:15.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b780b44c0 con 0x7f9b780a98f0 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 10 monclient: _finish_auth 0 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 10 monclient: _check_auth_tickets 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9b74001690 con 0x7f9b780a98f0 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b7cda8640 10 monclient: dump: 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:15.154 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:15.155 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:15.155 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:15.155 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:15.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b8000e640 5 monclient: authenticate success, global_id 4628 2024-03-25T18:37:15.155 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.152+0000 7f9b8000e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.158+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b740017f0 con 0x7f9b780a98f0 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.158+0000 7f9b7cda8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.158+0000 7f9b7cda8640 1 -- 172.21.15.55:0/1994982530 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9b74001c00 con 0x7f9b780a98f0 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.158+0000 7f9b8000e640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.158+0000 7f9b8000e640 4 set_mon_vals no callback set 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.158+0000 7f9b7cda8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.158+0000 7f9b7cda8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.158+0000 7f9b7cda8640 10 monclient: dump: 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:15.159 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:15.160 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:15.160 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:15.160 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:15.160 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:15.160 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:15.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.159+0000 7f9b8000e640 10 monclient: shutdown 2024-03-25T18:37:15.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.159+0000 7f9b8000e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:15.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.159+0000 7f9b8000e640 1 -- 172.21.15.55:0/1994982530 >> v1:172.21.15.55:6790/0 conn(0x7f9b780a98f0 legacy=0x7f9b780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.159+0000 7f9b8000e640 1 -- 172.21.15.55:0/1994982530 shutdown_connections 2024-03-25T18:37:15.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.159+0000 7f9b8000e640 1 -- 172.21.15.55:0/1994982530 >> 172.21.15.55:0/1994982530 conn(0x7f9b780b7ea0 msgr2=0x7f9b780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:15.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.159+0000 7f9b8000e640 1 -- 172.21.15.55:0/1994982530 shutdown_connections 2024-03-25T18:37:15.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.159+0000 7f9b8000e640 1 -- 172.21.15.55:0/1994982530 wait complete. 2024-03-25T18:37:15.162 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.160+0000 7f9b8000e640 10 monclient: build_initial_monmap 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.160+0000 7f9b8000e640 10 monclient: monmap: 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.160+0000 7f9b8000e640 1 librados: starting msgr at 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.160+0000 7f9b8000e640 1 librados: starting objecter 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.160+0000 7f9b8000e640 1 Processor -- start 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.160+0000 7f9b8000e640 1 -- start start 2024-03-25T18:37:15.163 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.160+0000 7f9b8000e640 1 librados: setting wanted keys 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.160+0000 7f9b8000e640 1 librados: calling monclient init 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.160+0000 7f9b8000e640 10 monclient: init 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 10 monclient(hunting): picked mon.noname-b con 0x7f9b780a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7813f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 10 monclient(hunting): picked mon.noname-a con 0x7f9b781430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78138b20 con 0x7f9b781430c0 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78137a80 con 0x7f9b780a98f0 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78139880 con 0x7f9b7813f710 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:15.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.161+0000 7f9b8000e640 10 monclient(hunting): authenticate will time out at 7839.418835s 2024-03-25T18:37:15.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.170+0000 7f9b7ddaa640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f9b780a98f0 0x7f9b78137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:41718/0 (socket says 172.21.15.55:41718) 2024-03-25T18:37:15.171 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.170+0000 7f9b7ddaa640 1 -- 172.21.15.55:0/320280533 learned_addr learned my addr 172.21.15.55:0/320280533 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:15.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.176+0000 7f9b6e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b780a98f0 2024-03-25T18:37:15.177 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.176+0000 7f9b6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:15.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.183+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403641514 0 0) 0x7f9b78137a80 con 0x7f9b780a98f0 2024-03-25T18:37:15.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.183+0000 7f9b6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:15.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.183+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:15.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.183+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:15.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.183+0000 7f9b6e7fc640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:37:15.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.183+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5c004eb0 con 0x7f9b780a98f0 2024-03-25T18:37:15.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.185+0000 7f9b6e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b7813f710 2024-03-25T18:37:15.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.185+0000 7f9b6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:15.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.193+0000 7f9b6e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b781430c0 2024-03-25T18:37:15.194 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.193+0000 7f9b6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:15.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.199+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3831940545 0 0) 0x7f9b5c004eb0 con 0x7f9b780a98f0 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.199+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b78137a80 con 0x7f9b780a98f0 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.199+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9b74001bc0 con 0x7f9b780a98f0 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.199+0000 7f9b6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.199+0000 7f9b6e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.199+0000 7f9b6e7fc640 10 monclient(hunting): dump: 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:15.201 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:15.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.199+0000 7f9b6e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:15.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.204+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088079865 0 0) 0x7f9b78139880 con 0x7f9b7813f710 2024-03-25T18:37:15.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.204+0000 7f9b6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:15.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.204+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:15.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.204+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:15.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.204+0000 7f9b6e7fc640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:37:15.205 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.204+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5c004eb0 con 0x7f9b7813f710 2024-03-25T18:37:15.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2854132003 0 0) 0x7f9b78137a80 con 0x7f9b780a98f0 2024-03-25T18:37:15.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 >> v1:172.21.15.55:6790/0 conn(0x7f9b7813f710 legacy=0x7f9b78139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.217 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 >> v1:172.21.15.55:6789/0 conn(0x7f9b781430c0 legacy=0x7f9b781e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 1 monclient: found mon.b 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b78139a10 con 0x7f9b780a98f0 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b8000e640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b8000e640 10 monclient: _renew_subs 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b8000e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b781dab00 con 0x7f9b780a98f0 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b8000e640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:15.218 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b8000e640 10 monclient: _renew_subs 2024-03-25T18:37:15.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b8000e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:15.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7813b4f0 con 0x7f9b780a98f0 2024-03-25T18:37:15.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.216+0000 7f9b8000e640 1 librados: init done 2024-03-25T18:37:15.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.217+0000 7f9b8000e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:15.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.217+0000 7f9b8000e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:15.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.217+0000 7f9b8000e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:15.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.217+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b780627b0 con 0x7f9b780a98f0 2024-03-25T18:37:15.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.484+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b740011f0 con 0x7f9b780a98f0 2024-03-25T18:37:15.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.484+0000 7f9b6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:15.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.484+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f9b74004c70 con 0x7f9b780a98f0 2024-03-25T18:37:15.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.484+0000 7f9b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:15.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.484+0000 7f9b6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:15.485 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.484+0000 7f9b6e7fc640 10 monclient: dump: 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:15.486 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.484+0000 7f9b6ffff640 4 set_mon_vals no callback set 2024-03-25T18:37:15.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.485+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f9b74004e90 con 0x7f9b780a98f0 2024-03-25T18:37:15.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.486+0000 7f9b7d5a9640 10 monclient: get_auth_request con 0x7f9b5c06dd40 auth_method 0 2024-03-25T18:37:15.487 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.486+0000 7f9b6e7fc640 10 client.4618.objecter ms_handle_connect 0x7f9b5c06dd40 2024-03-25T18:37:15.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.492+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f9b74054960 con 0x7f9b780a98f0 2024-03-25T18:37:15.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.492+0000 7f9b6e7fc640 10 client.4618.objecter ms_dispatch 0x7f9b78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:37:15.493 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.492+0000 7f9b6e7fc640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:37:15.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.492+0000 7f9b6e7fc640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:37:15.494 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.492+0000 7f9b6e7fc640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T18:37:15.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.497+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b740502a0 con 0x7f9b780a98f0 2024-03-25T18:37:15.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.497+0000 7f9b6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:15.498 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.497+0000 7f9b6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:15.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.765+0000 7f9b8000e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:15.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.765+0000 7f9b8000e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:15.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.765+0000 7f9b8000e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:15.766 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.765+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b780b3e80 con 0x7f9b780a98f0 2024-03-25T18:37:15.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.792+0000 7f9b6e7fc640 1 -- 172.21.15.55:0/320280533 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f9b740523a0 con 0x7f9b780a98f0 2024-03-25T18:37:15.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.792+0000 7f9b6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:15.793 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.792+0000 7f9b6e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:15.793 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:37:15.793 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.794+0000 7f9b8000e640 10 librados: watch_flush enter 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.794+0000 7f9b8000e640 10 librados: watch_flush exit 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.794+0000 7f9b8000e640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.794+0000 7f9b8000e640 10 client.4618.objecter successfully canceled tick 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.794+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 >> v1:172.21.15.55:6800/286536064 conn(0x7f9b5c06dd40 legacy=0x7f9b5c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.794+0000 7f9b8000e640 10 monclient: shutdown 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.794+0000 7f9b8000e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.794+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 >> v1:172.21.15.133:6789/0 conn(0x7f9b780a98f0 legacy=0x7f9b78137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.795+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 shutdown_connections 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.795+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 >> 172.21.15.55:0/320280533 conn(0x7f9b780b7ea0 msgr2=0x7f9b78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.795+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 shutdown_connections 2024-03-25T18:37:15.796 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.795+0000 7f9b8000e640 1 -- 172.21.15.55:0/320280533 wait complete. 2024-03-25T18:37:15.797 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:15.795+0000 7f9b8000e640 1 librados: shutdown 2024-03-25T18:37:15.808 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:16.450 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:26.620 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:16.501 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:26.671 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:16.527 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:26.697 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:16.554 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:26.691 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:16.564 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:09:26.734 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:16.567 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:26.705 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:16.573 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:09:26.743 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:16.599 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:26.737 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:16.603 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:09:26.773 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:22.614 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:22.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.729+0000 7f60cfd82640 10 monclient: get_monmap_and_config 2024-03-25T18:37:22.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.729+0000 7f60cfd82640 10 monclient: build_initial_monmap 2024-03-25T18:37:22.730 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.729+0000 7f60cfd82640 10 monclient: monmap: 2024-03-25T18:37:22.730 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:22.730 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:22.730 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:37:22.730076+0000 2024-03-25T18:37:22.730 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:37:22.730076+0000 2024-03-25T18:37:22.730 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:22.730 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:22.731 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:22.731 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:37:22.731 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:37:22.731 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:22.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.730+0000 7f60cfd82640 1 Processor -- start 2024-03-25T18:37:22.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.730+0000 7f60cfd82640 1 -- start start 2024-03-25T18:37:22.731 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.730+0000 7f60cfd82640 10 monclient: init 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 10 monclient(hunting): picked mon.noname-b con 0x7f60c8070410 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 10 monclient(hunting): picked mon.noname-a con 0x7f60c80707f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 10 monclient(hunting): picked mon.noname-c con 0x7f60c813db40 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c80adea0 con 0x7f60c80707f0 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c81445a0 con 0x7f60c813db40 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c80720a0 con 0x7f60c8070410 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:22.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.731+0000 7f60cfd82640 10 monclient(hunting): authenticate will time out at 7846.988627s 2024-03-25T18:37:22.736 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.735+0000 7f60cdb1e640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f60c8070410 0x7f60c8143e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:39362/0 (socket says 172.21.15.55:39362) 2024-03-25T18:37:22.736 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.735+0000 7f60cdb1e640 1 -- 172.21.15.55:0/2016390643 learned_addr learned my addr 172.21.15.55:0/2016390643 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:22.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.747+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840669263 0 0) 0x7f60c81445a0 con 0x7f60c813db40 2024-03-25T18:37:22.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.747+0000 7f60ccb1c640 10 monclient(hunting): _negotiate 2024-03-25T18:37:22.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.747+0000 7f60ccb1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:22.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.747+0000 7f60ccb1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:22.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.747+0000 7f60ccb1c640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:37:22.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.747+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b4003520 con 0x7f60c813db40 2024-03-25T18:37:22.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.748+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897551175 0 0) 0x7f60c80720a0 con 0x7f60c8070410 2024-03-25T18:37:22.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.748+0000 7f60ccb1c640 10 monclient(hunting): _negotiate 2024-03-25T18:37:22.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.748+0000 7f60ccb1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:22.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.748+0000 7f60ccb1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:22.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.748+0000 7f60ccb1c640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:37:22.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.748+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c81445a0 con 0x7f60c8070410 2024-03-25T18:37:22.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2677948400 0 0) 0x7f60c81445a0 con 0x7f60c8070410 2024-03-25T18:37:22.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 >> v1:172.21.15.133:6789/0 conn(0x7f60c813db40 legacy=0x7f60c814ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 >> v1:172.21.15.55:6789/0 conn(0x7f60c80707f0 legacy=0x7f60c813d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 10 monclient: _finish_hunting 0 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 1 monclient: found mon.noname-b 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8072380 con 0x7f60c8070410 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 10 monclient: _finish_auth 0 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 10 monclient: _check_auth_tickets 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f60c4001670 con 0x7f60c8070410 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60ccb1c640 10 monclient: dump: 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:22.763 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60cfd82640 5 monclient: authenticate success, global_id 4634 2024-03-25T18:37:22.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.761+0000 7f60cfd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.768+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60c40017d0 con 0x7f60c8070410 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.768+0000 7f60ccb1c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.768+0000 7f60ccb1c640 1 -- 172.21.15.55:0/2016390643 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f60c4001ba0 con 0x7f60c8070410 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.768+0000 7f60cfd82640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.768+0000 7f60cfd82640 4 set_mon_vals no callback set 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.768+0000 7f60ccb1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.768+0000 7f60ccb1c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.768+0000 7f60ccb1c640 10 monclient: dump: 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:22.769 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:22.770 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:22.770 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:22.770 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:22.770 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:22.770 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:22.770 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:22.770 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:22.770 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:22.771 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.769+0000 7f60cfd82640 10 monclient: shutdown 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.769+0000 7f60cfd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.769+0000 7f60cfd82640 1 -- 172.21.15.55:0/2016390643 >> v1:172.21.15.55:6790/0 conn(0x7f60c8070410 legacy=0x7f60c8143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.769+0000 7f60cfd82640 1 -- 172.21.15.55:0/2016390643 shutdown_connections 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.769+0000 7f60cfd82640 1 -- 172.21.15.55:0/2016390643 >> 172.21.15.55:0/2016390643 conn(0x7f60c806b630 msgr2=0x7f60c80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.769+0000 7f60cfd82640 1 -- 172.21.15.55:0/2016390643 shutdown_connections 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.769+0000 7f60cfd82640 1 -- 172.21.15.55:0/2016390643 wait complete. 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.770+0000 7f60cfd82640 10 monclient: build_initial_monmap 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.770+0000 7f60cfd82640 10 monclient: monmap: 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:22.772 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.770+0000 7f60cfd82640 1 librados: starting msgr at 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.770+0000 7f60cfd82640 1 librados: starting objecter 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.770+0000 7f60cfd82640 1 Processor -- start 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.770+0000 7f60cfd82640 1 -- start start 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.770+0000 7f60cfd82640 1 librados: setting wanted keys 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.770+0000 7f60cfd82640 1 librados: calling monclient init 2024-03-25T18:37:22.773 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.770+0000 7f60cfd82640 10 monclient: init 2024-03-25T18:37:22.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:22.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:37:22.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 10 monclient(hunting): picked mon.noname-c con 0x7f60c8070410 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:22.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 10 monclient(hunting): picked mon.noname-b con 0x7f60c80707f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:22.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 10 monclient(hunting): picked mon.noname-a con 0x7f60c813db40 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:22.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c80ac760 con 0x7f60c813db40 2024-03-25T18:37:22.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c8140f40 con 0x7f60c80707f0 2024-03-25T18:37:22.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c81410d0 con 0x7f60c8070410 2024-03-25T18:37:22.774 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:22.775 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.771+0000 7f60cfd82640 10 monclient(hunting): authenticate will time out at 7847.029079s 2024-03-25T18:37:22.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.782+0000 7f60cd31d640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f60c80707f0 0x7f60c81d6090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:42334/0 (socket says 172.21.15.55:42334) 2024-03-25T18:37:22.783 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.782+0000 7f60cd31d640 1 -- 172.21.15.55:0/4072897545 learned_addr learned my addr 172.21.15.55:0/4072897545 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:22.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.783+0000 7f60be7fc640 10 client.?.objecter ms_handle_connect 0x7f60c80707f0 2024-03-25T18:37:22.784 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.783+0000 7f60be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.790+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087916946 0 0) 0x7f60c8140f40 con 0x7f60c80707f0 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.790+0000 7f60be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.790+0000 7f60be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.790+0000 7f60be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.790+0000 7f60be7fc640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:37:22.791 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.790+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60ac004ee0 con 0x7f60c80707f0 2024-03-25T18:37:22.801 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.800+0000 7f60be7fc640 10 client.?.objecter ms_handle_connect 0x7f60c8070410 2024-03-25T18:37:22.801 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.800+0000 7f60be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:22.804 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.803+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2037429076 0 0) 0x7f60ac004ee0 con 0x7f60c80707f0 2024-03-25T18:37:22.804 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.803+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60c8140f40 con 0x7f60c80707f0 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.803+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f60b8001a00 con 0x7f60c80707f0 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.803+0000 7f60be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.803+0000 7f60be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.803+0000 7f60be7fc640 10 monclient(hunting): dump: 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:22.805 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:22.806 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.803+0000 7f60be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:22.807 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.806+0000 7f60be7fc640 10 client.?.objecter ms_handle_connect 0x7f60c813db40 2024-03-25T18:37:22.807 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.806+0000 7f60be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:22.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1255517504 0 0) 0x7f60c8140f40 con 0x7f60c80707f0 2024-03-25T18:37:22.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 >> v1:172.21.15.55:6790/0 conn(0x7f60c8070410 legacy=0x7f60c81d5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 >> v1:172.21.15.55:6789/0 conn(0x7f60c813db40 legacy=0x7f60c81e0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:22.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:22.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 1 monclient: found mon.b 2024-03-25T18:37:22.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:22.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:22.814 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8141260 con 0x7f60c80707f0 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60cfd82640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60cfd82640 10 monclient: _renew_subs 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60cfd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c80acc30 con 0x7f60c80707f0 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60cfd82640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60cfd82640 10 monclient: _renew_subs 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60cfd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c80ac140 con 0x7f60c80707f0 2024-03-25T18:37:22.815 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.813+0000 7f60cfd82640 1 librados: init done 2024-03-25T18:37:22.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.814+0000 7f60cfd82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:22.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.814+0000 7f60cfd82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:22.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.814+0000 7f60cfd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:22.816 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:22.814+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c80acfd0 con 0x7f60c80707f0 2024-03-25T18:37:23.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.101+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60b8001cc0 con 0x7f60c80707f0 2024-03-25T18:37:23.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.101+0000 7f60be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:23.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.101+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f60b8003a60 con 0x7f60c80707f0 2024-03-25T18:37:23.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.101+0000 7f60be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:23.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.101+0000 7f60be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:23.102 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.101+0000 7f60be7fc640 10 monclient: dump: 2024-03-25T18:37:23.102 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:23.103 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.101+0000 7f60bffff640 4 set_mon_vals no callback set 2024-03-25T18:37:23.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.102+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f60b8003c40 con 0x7f60c80707f0 2024-03-25T18:37:23.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.102+0000 7f60cdb1e640 10 monclient: get_auth_request con 0x7f60ac06dd10 auth_method 0 2024-03-25T18:37:23.104 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.102+0000 7f60be7fc640 10 client.4624.objecter ms_handle_connect 0x7f60ac06dd10 2024-03-25T18:37:23.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.104+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f60b8054930 con 0x7f60c80707f0 2024-03-25T18:37:23.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.104+0000 7f60be7fc640 10 client.4624.objecter ms_dispatch 0x7f60c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:37:23.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.104+0000 7f60be7fc640 3 client.4624.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:37:23.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.104+0000 7f60be7fc640 3 client.4624.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:37:23.105 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.104+0000 7f60be7fc640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-25T18:37:23.108 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.108+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60b8050270 con 0x7f60c80707f0 2024-03-25T18:37:23.109 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.108+0000 7f60be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:23.109 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.108+0000 7f60be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:23.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.336+0000 7f60cfd82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:23.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.336+0000 7f60cfd82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:23.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.336+0000 7f60cfd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:23.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.336+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60c80ac910 con 0x7f60c80707f0 2024-03-25T18:37:23.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.366+0000 7f60be7fc640 1 -- 172.21.15.55:0/4072897545 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f60b8052370 con 0x7f60c80707f0 2024-03-25T18:37:23.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.366+0000 7f60be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:23.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.366+0000 7f60be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:23.368 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:37:23.368 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:23.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.368+0000 7f60cfd82640 10 librados: watch_flush enter 2024-03-25T18:37:23.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.368+0000 7f60cfd82640 10 librados: watch_flush exit 2024-03-25T18:37:23.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.368+0000 7f60cfd82640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-25T18:37:23.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.368+0000 7f60cfd82640 10 client.4624.objecter successfully canceled tick 2024-03-25T18:37:23.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.368+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 >> v1:172.21.15.55:6800/286536064 conn(0x7f60ac06dd10 legacy=0x7f60ac0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:23.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.368+0000 7f60cfd82640 10 monclient: shutdown 2024-03-25T18:37:23.370 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.368+0000 7f60cfd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.368+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 >> v1:172.21.15.133:6789/0 conn(0x7f60c80707f0 legacy=0x7f60c81d6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.369+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 shutdown_connections 2024-03-25T18:37:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.369+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 >> 172.21.15.55:0/4072897545 conn(0x7f60c806b630 msgr2=0x7f60c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.369+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 shutdown_connections 2024-03-25T18:37:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.369+0000 7f60cfd82640 1 -- 172.21.15.55:0/4072897545 wait complete. 2024-03-25T18:37:23.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:23.369+0000 7f60cfd82640 1 librados: shutdown 2024-03-25T18:37:23.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:30.186 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:30.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.305+0000 7fcf6d4db640 10 monclient: get_monmap_and_config 2024-03-25T18:37:30.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.305+0000 7fcf6d4db640 10 monclient: build_initial_monmap 2024-03-25T18:37:30.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.305+0000 7fcf6d4db640 10 monclient: monmap: 2024-03-25T18:37:30.306 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:30.306 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:30.307 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:37:30.306249+0000 2024-03-25T18:37:30.307 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:37:30.306249+0000 2024-03-25T18:37:30.307 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:30.307 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:30.307 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:30.307 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:37:30.307 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:37:30.307 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:30.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.306+0000 7fcf6d4db640 1 Processor -- start 2024-03-25T18:37:30.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.306+0000 7fcf6d4db640 1 -- start start 2024-03-25T18:37:30.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.306+0000 7fcf6d4db640 10 monclient: init 2024-03-25T18:37:30.308 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.306+0000 7fcf6d4db640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:30.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.306+0000 7fcf6d4db640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:37:30.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.306+0000 7fcf6d4db640 10 monclient(hunting): picked mon.noname-c con 0x7fcf680a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:30.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.306+0000 7fcf6d4db640 10 monclient(hunting): picked mon.noname-b con 0x7fcf6813f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:30.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.307+0000 7fcf6d4db640 10 monclient(hunting): picked mon.noname-a con 0x7fcf681430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:30.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.307+0000 7fcf6d4db640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf68142520 con 0x7fcf681430c0 2024-03-25T18:37:30.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.307+0000 7fcf6d4db640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680b28a0 con 0x7fcf680a98f0 2024-03-25T18:37:30.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.307+0000 7fcf6d4db640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf680b2a30 con 0x7fcf6813f710 2024-03-25T18:37:30.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.307+0000 7fcf6d4db640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:30.309 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.307+0000 7fcf6d4db640 10 monclient(hunting): authenticate will time out at 7854.564489s 2024-03-25T18:37:30.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.319+0000 7fcf667fc640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fcf6813f710 0x7fcf681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:46732/0 (socket says 172.21.15.55:46732) 2024-03-25T18:37:30.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.319+0000 7fcf667fc640 1 -- 172.21.15.55:0/3187083095 learned_addr learned my addr 172.21.15.55:0/3187083095 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:30.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.341+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818522719 0 0) 0x7fcf680b2a30 con 0x7fcf6813f710 2024-03-25T18:37:30.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.341+0000 7fcf65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.341+0000 7fcf65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.341+0000 7fcf65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.341+0000 7fcf65ffb640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:37:30.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.341+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf50003520 con 0x7fcf6813f710 2024-03-25T18:37:30.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.342+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685423534 0 0) 0x7fcf680b28a0 con 0x7fcf680a98f0 2024-03-25T18:37:30.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.342+0000 7fcf65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.342+0000 7fcf65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.342+0000 7fcf65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.342+0000 7fcf65ffb640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:37:30.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.342+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf680b2a30 con 0x7fcf680a98f0 2024-03-25T18:37:30.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871048478 0 0) 0x7fcf50003520 con 0x7fcf6813f710 2024-03-25T18:37:30.353 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 >> v1:172.21.15.133:6789/0 conn(0x7fcf680a98f0 legacy=0x7fcf680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 >> v1:172.21.15.55:6789/0 conn(0x7fcf681430c0 legacy=0x7fcf6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 1 monclient: found mon.noname-b 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf680b44c0 con 0x7fcf6813f710 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 10 monclient: _finish_auth 0 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcf5c001690 con 0x7fcf6813f710 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf65ffb640 10 monclient: dump: 2024-03-25T18:37:30.354 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf6d4db640 5 monclient: authenticate success, global_id 4640 2024-03-25T18:37:30.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.352+0000 7fcf6d4db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:30.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.361+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf5c0017f0 con 0x7fcf6813f710 2024-03-25T18:37:30.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.361+0000 7fcf65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:30.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.361+0000 7fcf65ffb640 1 -- 172.21.15.55:0/3187083095 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcf5c001c00 con 0x7fcf6813f710 2024-03-25T18:37:30.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.361+0000 7fcf6d4db640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:30.362 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.361+0000 7fcf6d4db640 4 set_mon_vals no callback set 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.361+0000 7fcf65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.361+0000 7fcf65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.361+0000 7fcf65ffb640 10 monclient: dump: 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:30.363 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.362+0000 7fcf6d4db640 10 monclient: shutdown 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.362+0000 7fcf6d4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.362+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3187083095 >> v1:172.21.15.55:6790/0 conn(0x7fcf6813f710 legacy=0x7fcf681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.362+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3187083095 shutdown_connections 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.362+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3187083095 >> 172.21.15.55:0/3187083095 conn(0x7fcf680b7ea0 msgr2=0x7fcf680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.362+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3187083095 shutdown_connections 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.362+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3187083095 wait complete. 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.363+0000 7fcf6d4db640 10 monclient: build_initial_monmap 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.363+0000 7fcf6d4db640 10 monclient: monmap: 2024-03-25T18:37:30.365 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.363+0000 7fcf6d4db640 1 librados: starting msgr at 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.363+0000 7fcf6d4db640 1 librados: starting objecter 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.364+0000 7fcf6d4db640 1 Processor -- start 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.364+0000 7fcf6d4db640 1 -- start start 2024-03-25T18:37:30.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.364+0000 7fcf6d4db640 1 librados: setting wanted keys 2024-03-25T18:37:30.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.364+0000 7fcf6d4db640 1 librados: calling monclient init 2024-03-25T18:37:30.367 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.364+0000 7fcf6d4db640 10 monclient: init 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.364+0000 7fcf6d4db640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.365+0000 7fcf6d4db640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.365+0000 7fcf6d4db640 10 monclient(hunting): picked mon.noname-c con 0x7fcf680a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.365+0000 7fcf6d4db640 10 monclient(hunting): picked mon.noname-a con 0x7fcf6813f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.365+0000 7fcf6d4db640 10 monclient(hunting): picked mon.noname-b con 0x7fcf681430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.365+0000 7fcf6d4db640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf681370a0 con 0x7fcf6813f710 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.365+0000 7fcf6d4db640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf68135090 con 0x7fcf681430c0 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.365+0000 7fcf6d4db640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf68135220 con 0x7fcf680a98f0 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.365+0000 7fcf6d4db640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:30.368 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.365+0000 7fcf6d4db640 10 monclient(hunting): authenticate will time out at 7854.622687s 2024-03-25T18:37:30.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.377+0000 7fcf667fc640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fcf6813f710 0x7fcf681d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:43594/0 (socket says 172.21.15.55:43594) 2024-03-25T18:37:30.378 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.377+0000 7fcf667fc640 1 -- 172.21.15.55:0/3324648615 learned_addr learned my addr 172.21.15.55:0/3324648615 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:30.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.379+0000 7fcf4f7fe640 10 client.?.objecter ms_handle_connect 0x7fcf681430c0 2024-03-25T18:37:30.380 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.379+0000 7fcf4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:30.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.382+0000 7fcf4f7fe640 10 client.?.objecter ms_handle_connect 0x7fcf6813f710 2024-03-25T18:37:30.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.382+0000 7fcf4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:30.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.382+0000 7fcf4f7fe640 10 client.?.objecter ms_handle_connect 0x7fcf680a98f0 2024-03-25T18:37:30.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.382+0000 7fcf4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:30.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.391+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145793123 0 0) 0x7fcf68135090 con 0x7fcf681430c0 2024-03-25T18:37:30.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.391+0000 7fcf4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.391+0000 7fcf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.391+0000 7fcf4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.391+0000 7fcf4f7fe640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:37:30.392 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.391+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf54004eb0 con 0x7fcf681430c0 2024-03-25T18:37:30.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.401+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909206520 0 0) 0x7fcf68135220 con 0x7fcf680a98f0 2024-03-25T18:37:30.402 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.401+0000 7fcf4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.401+0000 7fcf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.401+0000 7fcf4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.401+0000 7fcf4f7fe640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:37:30.403 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.401+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf68135090 con 0x7fcf680a98f0 2024-03-25T18:37:30.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.403+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112659176 0 0) 0x7fcf681370a0 con 0x7fcf6813f710 2024-03-25T18:37:30.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.403+0000 7fcf4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:37:30.404 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.403+0000 7fcf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:30.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.403+0000 7fcf4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:30.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.403+0000 7fcf4f7fe640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:37:30.405 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.403+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf68135220 con 0x7fcf6813f710 2024-03-25T18:37:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.411+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1045535522 0 0) 0x7fcf54004eb0 con 0x7fcf681430c0 2024-03-25T18:37:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.411+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf681370a0 con 0x7fcf681430c0 2024-03-25T18:37:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.411+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcf60001b20 con 0x7fcf681430c0 2024-03-25T18:37:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.411+0000 7fcf4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.411+0000 7fcf4f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:30.412 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.411+0000 7fcf4f7fe640 10 monclient(hunting): dump: 2024-03-25T18:37:30.412 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:30.412 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:30.412 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:30.413 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:30.413 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:30.413 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:30.413 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:30.413 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:30.413 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:30.413 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:30.413 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.411+0000 7fcf4f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2887190352 0 0) 0x7fcf681370a0 con 0x7fcf681430c0 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 >> v1:172.21.15.55:6790/0 conn(0x7fcf680a98f0 legacy=0x7fcf681dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 >> v1:172.21.15.55:6789/0 conn(0x7fcf6813f710 legacy=0x7fcf681d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:37:30.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 1 monclient: found mon.b 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf681d5000 con 0x7fcf681430c0 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf6d4db640 5 monclient: authenticate success, global_id 4630 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf6d4db640 10 monclient: _renew_subs 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf6d4db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf681d52b0 con 0x7fcf681430c0 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf6d4db640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf6d4db640 10 monclient: _renew_subs 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf6d4db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:30.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf68135910 con 0x7fcf681430c0 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.426+0000 7fcf6d4db640 1 librados: init done 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.427+0000 7fcf6d4db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.427+0000 7fcf6d4db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.427+0000 7fcf6d4db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:30.429 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.427+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf680627b0 con 0x7fcf681430c0 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.709+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf60002030 con 0x7fcf681430c0 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.709+0000 7fcf4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.709+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fcf60003b10 con 0x7fcf681430c0 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.709+0000 7fcf4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.709+0000 7fcf4f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.709+0000 7fcf4f7fe640 10 monclient: dump: 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:30.711 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:30.712 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:30.712 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:30.712 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:30.712 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:30.712 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:30.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.709+0000 7fcf657fa640 4 set_mon_vals no callback set 2024-03-25T18:37:30.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.711+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fcf60003d30 con 0x7fcf681430c0 2024-03-25T18:37:30.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.711+0000 7fcf66ffd640 10 monclient: get_auth_request con 0x7fcf5406dd50 auth_method 0 2024-03-25T18:37:30.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.711+0000 7fcf4f7fe640 10 client.4630.objecter ms_handle_connect 0x7fcf5406dd50 2024-03-25T18:37:30.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.716+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fcf600538f0 con 0x7fcf681430c0 2024-03-25T18:37:30.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.716+0000 7fcf4f7fe640 10 client.4630.objecter ms_dispatch 0x7fcf68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:37:30.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.716+0000 7fcf4f7fe640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:37:30.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.716+0000 7fcf4f7fe640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:37:30.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.716+0000 7fcf4f7fe640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T18:37:30.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.716+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf600b08b0 con 0x7fcf681430c0 2024-03-25T18:37:30.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.716+0000 7fcf4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:30.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.716+0000 7fcf4f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:30.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.957+0000 7fcf6d4db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:30.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.957+0000 7fcf6d4db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:30.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.957+0000 7fcf6d4db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:30.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.957+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf680008d0 con 0x7fcf681430c0 2024-03-25T18:37:30.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.976+0000 7fcf4f7fe640 1 -- 172.21.15.55:0/3324648615 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7fcf6004f230 con 0x7fcf681430c0 2024-03-25T18:37:30.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.976+0000 7fcf4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:30.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.976+0000 7fcf4f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:30.977 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:37:30.977 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:30.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.978+0000 7fcf6d4db640 10 librados: watch_flush enter 2024-03-25T18:37:30.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.978+0000 7fcf6d4db640 10 librados: watch_flush exit 2024-03-25T18:37:30.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.978+0000 7fcf6d4db640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T18:37:30.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.978+0000 7fcf6d4db640 10 client.4630.objecter successfully canceled tick 2024-03-25T18:37:30.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.978+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 >> v1:172.21.15.55:6800/286536064 conn(0x7fcf5406dd50 legacy=0x7fcf54070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.978+0000 7fcf6d4db640 10 monclient: shutdown 2024-03-25T18:37:30.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.979+0000 7fcf6d4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:30.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.979+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 >> v1:172.21.15.133:6789/0 conn(0x7fcf681430c0 legacy=0x7fcf681d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:30.980 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.979+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 shutdown_connections 2024-03-25T18:37:30.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.979+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 >> 172.21.15.55:0/3324648615 conn(0x7fcf680b7ea0 msgr2=0x7fcf68074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:30.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.979+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 shutdown_connections 2024-03-25T18:37:30.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.979+0000 7fcf6d4db640 1 -- 172.21.15.55:0/3324648615 wait complete. 2024-03-25T18:37:30.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:30.979+0000 7fcf6d4db640 1 librados: shutdown 2024-03-25T18:37:30.993 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:36.556 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:46.726 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.568 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:46.738 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.596 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:09:46.766 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.627 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:46.765 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.632 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:09:46.803 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.640 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:46.777 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.644 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:09:46.814 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.668 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:09:46.805 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.672 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:09:46.842 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:37.799 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:37.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.915+0000 7f3793e63640 10 monclient: get_monmap_and_config 2024-03-25T18:37:37.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.915+0000 7f3793e63640 10 monclient: build_initial_monmap 2024-03-25T18:37:37.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.915+0000 7f3793e63640 10 monclient: monmap: 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:37:37.916248+0000 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:37:37.916248+0000 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:37:37.917 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:37.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.916+0000 7f3793e63640 1 Processor -- start 2024-03-25T18:37:37.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.916+0000 7f3793e63640 1 -- start start 2024-03-25T18:37:37.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.916+0000 7f3793e63640 10 monclient: init 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 10 monclient(hunting): picked mon.noname-b con 0x7f378c0a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 10 monclient(hunting): picked mon.noname-a con 0x7f378c13f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 10 monclient(hunting): picked mon.noname-c con 0x7f378c1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c142520 con 0x7f378c13f710 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c0b28a0 con 0x7f378c1430c0 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c0b2a30 con 0x7f378c0a98f0 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:37.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.917+0000 7f3793e63640 10 monclient(hunting): authenticate will time out at 7862.174922s 2024-03-25T18:37:37.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.923+0000 7f3792400640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f378c1430c0 0x7f378c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:44832/0 (socket says 172.21.15.55:44832) 2024-03-25T18:37:37.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.923+0000 7f3792400640 1 -- 172.21.15.55:0/636447192 learned_addr learned my addr 172.21.15.55:0/636447192 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:37.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.935+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108358297 0 0) 0x7f378c142520 con 0x7f378c13f710 2024-03-25T18:37:37.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.935+0000 7f3790bfd640 10 monclient(hunting): _negotiate 2024-03-25T18:37:37.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.935+0000 7f3790bfd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:37.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.935+0000 7f3790bfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:37.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.935+0000 7f3790bfd640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:37:37.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.935+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3778003520 con 0x7f378c13f710 2024-03-25T18:37:37.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.938+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098857117 0 0) 0x7f378c0b28a0 con 0x7f378c1430c0 2024-03-25T18:37:37.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.938+0000 7f3790bfd640 10 monclient(hunting): _negotiate 2024-03-25T18:37:37.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.938+0000 7f3790bfd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:37.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.938+0000 7f3790bfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:37.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.938+0000 7f3790bfd640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:37:37.940 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.938+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378c142520 con 0x7f378c1430c0 2024-03-25T18:37:37.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2214533444 0 0) 0x7f3778003520 con 0x7f378c13f710 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 >> v1:172.21.15.55:6790/0 conn(0x7f378c0a98f0 legacy=0x7f378c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 >> v1:172.21.15.133:6789/0 conn(0x7f378c1430c0 legacy=0x7f378c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 10 monclient: _finish_hunting 0 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 1 monclient: found mon.noname-a 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c0b44c0 con 0x7f378c13f710 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 10 monclient: _finish_auth 0 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 10 monclient: _check_auth_tickets 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f377c001690 con 0x7f378c13f710 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:37:37.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3790bfd640 10 monclient: dump: 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3793e63640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:37:37.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.947+0000 7f3793e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.955+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f377c0017f0 con 0x7f378c13f710 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.955+0000 7f3790bfd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.955+0000 7f3790bfd640 1 -- 172.21.15.55:0/636447192 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f377c001c00 con 0x7f378c13f710 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.955+0000 7f3790bfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.955+0000 7f3790bfd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.955+0000 7f3790bfd640 10 monclient: dump: 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:37.956 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:37.957 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:37.957 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:37.957 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:37.957 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:37.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.955+0000 7f3793e63640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:37.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.955+0000 7f3793e63640 4 set_mon_vals no callback set 2024-03-25T18:37:37.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.956+0000 7f3793e63640 10 monclient: shutdown 2024-03-25T18:37:37.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.956+0000 7f3793e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:37.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.956+0000 7f3793e63640 1 -- 172.21.15.55:0/636447192 >> v1:172.21.15.55:6789/0 conn(0x7f378c13f710 legacy=0x7f378c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:37.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.956+0000 7f3793e63640 1 -- 172.21.15.55:0/636447192 shutdown_connections 2024-03-25T18:37:37.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.956+0000 7f3793e63640 1 -- 172.21.15.55:0/636447192 >> 172.21.15.55:0/636447192 conn(0x7f378c0b7ea0 msgr2=0x7f378c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.956+0000 7f3793e63640 1 -- 172.21.15.55:0/636447192 shutdown_connections 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.956+0000 7f3793e63640 1 -- 172.21.15.55:0/636447192 wait complete. 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.957+0000 7f3793e63640 10 monclient: build_initial_monmap 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.957+0000 7f3793e63640 10 monclient: monmap: 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:37:37.960 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.957+0000 7f3793e63640 1 librados: starting msgr at 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.957+0000 7f3793e63640 1 librados: starting objecter 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.957+0000 7f3793e63640 1 Processor -- start 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.957+0000 7f3793e63640 1 -- start start 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.957+0000 7f3793e63640 1 librados: setting wanted keys 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.957+0000 7f3793e63640 1 librados: calling monclient init 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.957+0000 7f3793e63640 10 monclient: init 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 10 monclient(hunting): picked mon.noname-b con 0x7f378c0a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 10 monclient(hunting): picked mon.noname-a con 0x7f378c13f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 10 monclient(hunting): picked mon.noname-c con 0x7f378c1430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c0abd70 con 0x7f378c13f710 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c1d7710 con 0x7f378c0a98f0 2024-03-25T18:37:37.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378c1d8340 con 0x7f378c1430c0 2024-03-25T18:37:37.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:37.962 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.958+0000 7f3793e63640 10 monclient(hunting): authenticate will time out at 7862.215646s 2024-03-25T18:37:37.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.980+0000 7f3791bff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f378c0a98f0 0x7f378c137be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:44872/0 (socket says 172.21.15.55:44872) 2024-03-25T18:37:37.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.980+0000 7f3791bff640 1 -- 172.21.15.55:0/1238531702 learned_addr learned my addr 172.21.15.55:0/1238531702 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:37.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.981+0000 7f37827fc640 10 client.?.objecter ms_handle_connect 0x7f378c0a98f0 2024-03-25T18:37:37.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.981+0000 7f37827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:37.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.989+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977418845 0 0) 0x7f378c1d7710 con 0x7f378c0a98f0 2024-03-25T18:37:37.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.989+0000 7f37827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:37.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.989+0000 7f37827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:37.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.989+0000 7f37827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:37.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.989+0000 7f37827fc640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:37:37.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.989+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3770004eb0 con 0x7f378c0a98f0 2024-03-25T18:37:37.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.992+0000 7f37827fc640 10 client.?.objecter ms_handle_connect 0x7f378c13f710 2024-03-25T18:37:37.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.992+0000 7f37827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:37.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.997+0000 7f37827fc640 10 client.?.objecter ms_handle_connect 0x7f378c1430c0 2024-03-25T18:37:37.997 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:37.997+0000 7f37827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:38.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.005+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 960964620 0 0) 0x7f3770004eb0 con 0x7f378c0a98f0 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.006+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f378c1d7710 con 0x7f378c0a98f0 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.006+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3788001a30 con 0x7f378c0a98f0 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.006+0000 7f37827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.006+0000 7f37827fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.006+0000 7f37827fc640 10 monclient(hunting): dump: 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:38.007 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:38.008 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.006+0000 7f37827fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:38.017 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.016+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568861936 0 0) 0x7f378c1d8340 con 0x7f378c1430c0 2024-03-25T18:37:38.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.017+0000 7f37827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:38.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.017+0000 7f37827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:38.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.017+0000 7f37827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:38.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.017+0000 7f37827fc640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:37:38.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.017+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3770004eb0 con 0x7f378c1430c0 2024-03-25T18:37:38.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.025+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035918074 0 0) 0x7f378c0abd70 con 0x7f378c13f710 2024-03-25T18:37:38.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.025+0000 7f37827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:38.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.025+0000 7f37827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:38.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.025+0000 7f37827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:38.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.025+0000 7f37827fc640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:37:38.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.025+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378c1d8340 con 0x7f378c13f710 2024-03-25T18:37:38.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1236589827 0 0) 0x7f378c1d7710 con 0x7f378c0a98f0 2024-03-25T18:37:38.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 >> v1:172.21.15.55:6790/0 conn(0x7f378c1430c0 legacy=0x7f378c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 >> v1:172.21.15.55:6789/0 conn(0x7f378c13f710 legacy=0x7f378c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:38.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 1 monclient: found mon.b 2024-03-25T18:37:38.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:38.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:38.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c1d84d0 con 0x7f378c0a98f0 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f37827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f3793e63640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f3793e63640 10 monclient: _renew_subs 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.032+0000 7f3793e63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.033+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378c1d41e0 con 0x7f378c0a98f0 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.033+0000 7f3793e63640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.033+0000 7f3793e63640 10 monclient: _renew_subs 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.033+0000 7f3793e63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.033+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f378c1d5150 con 0x7f378c0a98f0 2024-03-25T18:37:38.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.033+0000 7f3793e63640 1 librados: init done 2024-03-25T18:37:38.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.034+0000 7f3793e63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:38.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.034+0000 7f3793e63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:38.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.034+0000 7f3793e63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:38.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.034+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378c0627b0 con 0x7f378c0a98f0 2024-03-25T18:37:38.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.329+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3788001cb0 con 0x7f378c0a98f0 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.329+0000 7f37827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.329+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f3788003a60 con 0x7f378c0a98f0 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.329+0000 7f37827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.329+0000 7f37827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.329+0000 7f37827fc640 10 monclient: dump: 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:38.331 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:38.332 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:38.332 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:38.332 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:38.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.329+0000 7f3783fff640 4 set_mon_vals no callback set 2024-03-25T18:37:38.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.330+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f3788003c40 con 0x7f378c0a98f0 2024-03-25T18:37:38.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.331+0000 7f37913fe640 10 monclient: get_auth_request con 0x7f377006dea0 auth_method 0 2024-03-25T18:37:38.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.331+0000 7f37827fc640 10 client.4636.objecter ms_handle_connect 0x7f377006dea0 2024-03-25T18:37:38.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.335+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f3788054870 con 0x7f378c0a98f0 2024-03-25T18:37:38.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.335+0000 7f37827fc640 10 client.4636.objecter ms_dispatch 0x7f378c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:37:38.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.335+0000 7f37827fc640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:37:38.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.335+0000 7f37827fc640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:37:38.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.335+0000 7f37827fc640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T18:37:38.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.335+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3788050380 con 0x7f378c0a98f0 2024-03-25T18:37:38.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.335+0000 7f37827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:38.337 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.335+0000 7f37827fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:38.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.596+0000 7f3793e63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:38.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.596+0000 7f3793e63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:38.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.596+0000 7f3793e63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:38.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.596+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f378c1d78a0 con 0x7f378c0a98f0 2024-03-25T18:37:38.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.646+0000 7f37827fc640 1 -- 172.21.15.55:0/1238531702 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f37880522b0 con 0x7f378c0a98f0 2024-03-25T18:37:38.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.646+0000 7f37827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:38.647 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.646+0000 7f37827fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:38.647 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:37:38.647 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:38.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.647+0000 7f3793e63640 10 librados: watch_flush enter 2024-03-25T18:37:38.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 10 librados: watch_flush exit 2024-03-25T18:37:38.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T18:37:38.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 10 client.4636.objecter successfully canceled tick 2024-03-25T18:37:38.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 >> v1:172.21.15.55:6800/286536064 conn(0x7f377006dea0 legacy=0x7f3770070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 10 monclient: shutdown 2024-03-25T18:37:38.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:38.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 >> v1:172.21.15.133:6789/0 conn(0x7f378c0a98f0 legacy=0x7f378c137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:38.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 shutdown_connections 2024-03-25T18:37:38.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 >> 172.21.15.55:0/1238531702 conn(0x7f378c0b7ea0 msgr2=0x7f378c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:38.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 shutdown_connections 2024-03-25T18:37:38.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 1 -- 172.21.15.55:0/1238531702 wait complete. 2024-03-25T18:37:38.650 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:38.648+0000 7f3793e63640 1 librados: shutdown 2024-03-25T18:37:38.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:45.468 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:45.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.587+0000 7f045bf8d640 10 monclient: get_monmap_and_config 2024-03-25T18:37:45.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.587+0000 7f045bf8d640 10 monclient: build_initial_monmap 2024-03-25T18:37:45.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.587+0000 7f045bf8d640 10 monclient: monmap: 2024-03-25T18:37:45.589 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:45.590 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:45.590 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:37:45.589592+0000 2024-03-25T18:37:45.590 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:37:45.589592+0000 2024-03-25T18:37:45.590 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:45.590 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:45.590 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:45.590 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:37:45.590 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:37:45.590 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:45.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.588+0000 7f045bf8d640 1 Processor -- start 2024-03-25T18:37:45.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.588+0000 7f045bf8d640 1 -- start start 2024-03-25T18:37:45.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.588+0000 7f045bf8d640 10 monclient: init 2024-03-25T18:37:45.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.589+0000 7f045bf8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:45.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.589+0000 7f045bf8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:45.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.589+0000 7f045bf8d640 10 monclient(hunting): picked mon.noname-b con 0x7f04540a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:45.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.589+0000 7f045bf8d640 10 monclient(hunting): picked mon.noname-a con 0x7f045413f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:45.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.589+0000 7f045bf8d640 10 monclient(hunting): picked mon.noname-c con 0x7f04541430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:45.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.589+0000 7f045bf8d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0454142520 con 0x7f045413f710 2024-03-25T18:37:45.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.589+0000 7f045bf8d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04540b28a0 con 0x7f04541430c0 2024-03-25T18:37:45.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.589+0000 7f045bf8d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04540b2a30 con 0x7f04540a98f0 2024-03-25T18:37:45.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.589+0000 7f045bf8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:45.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.590+0000 7f045bf8d640 10 monclient(hunting): authenticate will time out at 7869.848365s 2024-03-25T18:37:45.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.594+0000 7f0459d29640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f04540a98f0 0x7f04540a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:47710/0 (socket says 172.21.15.55:47710) 2024-03-25T18:37:45.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.594+0000 7f0459d29640 1 -- 172.21.15.55:0/2448550921 learned_addr learned my addr 172.21.15.55:0/2448550921 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.612+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116293097 0 0) 0x7f04540b2a30 con 0x7f04540a98f0 2024-03-25T18:37:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.613+0000 7f0458d27640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.613+0000 7f0458d27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.613+0000 7f0458d27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.613+0000 7f0458d27640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:37:45.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.613+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0440003520 con 0x7f04540a98f0 2024-03-25T18:37:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.616+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934681443 0 0) 0x7f04540b28a0 con 0x7f04541430c0 2024-03-25T18:37:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.616+0000 7f0458d27640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.616+0000 7f0458d27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.616+0000 7f0458d27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.616+0000 7f0458d27640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:37:45.618 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.616+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04540b2a30 con 0x7f04541430c0 2024-03-25T18:37:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.617+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16399665 0 0) 0x7f0454142520 con 0x7f045413f710 2024-03-25T18:37:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.617+0000 7f0458d27640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.617+0000 7f0458d27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.617+0000 7f0458d27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.617+0000 7f0458d27640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:37:45.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.617+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04540b28a0 con 0x7f045413f710 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2381806938 0 0) 0x7f04540b2a30 con 0x7f04541430c0 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 >> v1:172.21.15.55:6790/0 conn(0x7f04540a98f0 legacy=0x7f04540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 >> v1:172.21.15.55:6789/0 conn(0x7f045413f710 legacy=0x7f04541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 10 monclient: _finish_hunting 0 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 1 monclient: found mon.noname-c 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04540b44c0 con 0x7f04541430c0 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 10 monclient: _finish_auth 0 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 10 monclient: _check_auth_tickets 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f044c0019f0 con 0x7f04541430c0 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f045bf8d640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:37:45.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f045bf8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f0458d27640 10 monclient: dump: 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:45.624 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.620+0000 7f045bf8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.635+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f044c001b50 con 0x7f04541430c0 2024-03-25T18:37:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.635+0000 7f0458d27640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.635+0000 7f0458d27640 1 -- 172.21.15.55:0/2448550921 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f044c001f60 con 0x7f04541430c0 2024-03-25T18:37:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.635+0000 7f045bf8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.635+0000 7f045bf8d640 4 set_mon_vals no callback set 2024-03-25T18:37:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.635+0000 7f0458d27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:45.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.635+0000 7f0458d27640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.635+0000 7f0458d27640 10 monclient: dump: 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:45.638 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:45.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.636+0000 7f045bf8d640 10 monclient: shutdown 2024-03-25T18:37:45.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.636+0000 7f045bf8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:45.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.636+0000 7f045bf8d640 1 -- 172.21.15.55:0/2448550921 >> v1:172.21.15.133:6789/0 conn(0x7f04541430c0 legacy=0x7f045414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.636+0000 7f045bf8d640 1 -- 172.21.15.55:0/2448550921 shutdown_connections 2024-03-25T18:37:45.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.636+0000 7f045bf8d640 1 -- 172.21.15.55:0/2448550921 >> 172.21.15.55:0/2448550921 conn(0x7f04540b7ea0 msgr2=0x7f04540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:45.640 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.636+0000 7f045bf8d640 1 -- 172.21.15.55:0/2448550921 shutdown_connections 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.636+0000 7f045bf8d640 1 -- 172.21.15.55:0/2448550921 wait complete. 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.637+0000 7f045bf8d640 10 monclient: build_initial_monmap 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.637+0000 7f045bf8d640 10 monclient: monmap: 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.637+0000 7f045bf8d640 1 librados: starting msgr at 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.637+0000 7f045bf8d640 1 librados: starting objecter 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.637+0000 7f045bf8d640 1 Processor -- start 2024-03-25T18:37:45.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.637+0000 7f045bf8d640 1 -- start start 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.637+0000 7f045bf8d640 1 librados: setting wanted keys 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.637+0000 7f045bf8d640 1 librados: calling monclient init 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.637+0000 7f045bf8d640 10 monclient: init 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 10 monclient(hunting): picked mon.noname-a con 0x7f04540a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 10 monclient(hunting): picked mon.noname-b con 0x7f045413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 10 monclient(hunting): picked mon.noname-c con 0x7f04541430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04540b71e0 con 0x7f04540a98f0 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045413c780 con 0x7f045413f710 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0454129440 con 0x7f04541430c0 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:45.642 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.638+0000 7f045bf8d640 10 monclient(hunting): authenticate will time out at 7869.897074s 2024-03-25T18:37:45.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.646+0000 7f045a52a640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f04541430c0 0x7f045412a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:47746/0 (socket says 172.21.15.55:47746) 2024-03-25T18:37:45.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.646+0000 7f045a52a640 1 -- 172.21.15.55:0/2709033034 learned_addr learned my addr 172.21.15.55:0/2709033034 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:45.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.646+0000 7f044a7fc640 10 client.?.objecter ms_handle_connect 0x7f04541430c0 2024-03-25T18:37:45.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.647+0000 7f044a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:45.656 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.655+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495400545 0 0) 0x7f0454129440 con 0x7f04541430c0 2024-03-25T18:37:45.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.655+0000 7f044a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.655+0000 7f044a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.655+0000 7f044a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.655+0000 7f044a7fc640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:37:45.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.655+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0438004eb0 con 0x7f04541430c0 2024-03-25T18:37:45.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.661+0000 7f044a7fc640 10 client.?.objecter ms_handle_connect 0x7f045413f710 2024-03-25T18:37:45.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.661+0000 7f044a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:45.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.663+0000 7f044a7fc640 10 client.?.objecter ms_handle_connect 0x7f04540a98f0 2024-03-25T18:37:45.665 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.663+0000 7f044a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:45.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.665+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3943900528 0 0) 0x7f0438004eb0 con 0x7f04541430c0 2024-03-25T18:37:45.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.665+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0454129440 con 0x7f04541430c0 2024-03-25T18:37:45.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.665+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f044c001fa0 con 0x7f04541430c0 2024-03-25T18:37:45.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.665+0000 7f044a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:45.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.665+0000 7f044a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:45.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.665+0000 7f044a7fc640 10 monclient(hunting): dump: 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:45.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.665+0000 7f044a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:45.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.674+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835936203 0 0) 0x7f045413c780 con 0x7f045413f710 2024-03-25T18:37:45.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.674+0000 7f044a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.674+0000 7f044a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.674+0000 7f044a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.674+0000 7f044a7fc640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:37:45.676 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.674+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0438004eb0 con 0x7f045413f710 2024-03-25T18:37:45.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.678+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939539742 0 0) 0x7f04540b71e0 con 0x7f04540a98f0 2024-03-25T18:37:45.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.678+0000 7f044a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:45.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.678+0000 7f044a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:45.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.678+0000 7f044a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:45.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.678+0000 7f044a7fc640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:37:45.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.678+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045413c780 con 0x7f04540a98f0 2024-03-25T18:37:45.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 418735224 0 0) 0x7f0454129440 con 0x7f04541430c0 2024-03-25T18:37:45.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 >> v1:172.21.15.133:6789/0 conn(0x7f045413f710 legacy=0x7f0454129e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.683 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 >> v1:172.21.15.55:6789/0 conn(0x7f04540a98f0 legacy=0x7f045413c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:45.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:45.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 1 monclient: found mon.c 2024-03-25T18:37:45.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:45.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:37:45.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0454129720 con 0x7f04541430c0 2024-03-25T18:37:45.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:45.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:45.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f044a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:45.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f045bf8d640 5 monclient: authenticate success, global_id 4655 2024-03-25T18:37:45.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f045bf8d640 10 monclient: _renew_subs 2024-03-25T18:37:45.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f045bf8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:37:45.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04540b7c10 con 0x7f04541430c0 2024-03-25T18:37:45.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f045bf8d640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:45.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f045bf8d640 10 monclient: _renew_subs 2024-03-25T18:37:45.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f045bf8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:37:45.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0454128500 con 0x7f04541430c0 2024-03-25T18:37:45.686 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.681+0000 7f045bf8d640 1 librados: init done 2024-03-25T18:37:45.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.682+0000 7f045bf8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:45.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.682+0000 7f045bf8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:45.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.682+0000 7f045bf8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:37:45.687 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:45.682+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04540b3ef0 con 0x7f04541430c0 2024-03-25T18:37:46.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.011+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f044c003c20 con 0x7f04541430c0 2024-03-25T18:37:46.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.011+0000 7f044a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:46.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.011+0000 7f044bfff640 4 set_mon_vals no callback set 2024-03-25T18:37:46.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.011+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f044c003fb0 con 0x7f04541430c0 2024-03-25T18:37:46.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.011+0000 7f044a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:46.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.011+0000 7f044a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:46.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.011+0000 7f044a7fc640 10 monclient: dump: 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:46.014 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:46.015 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.012+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f044c0041d0 con 0x7f04541430c0 2024-03-25T18:37:46.015 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.013+0000 7f0459d29640 10 monclient: get_auth_request con 0x7f043806dea0 auth_method 0 2024-03-25T18:37:46.015 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.013+0000 7f044a7fc640 10 client.4655.objecter ms_handle_connect 0x7f043806dea0 2024-03-25T18:37:46.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.019+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f044c054ba0 con 0x7f04541430c0 2024-03-25T18:37:46.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.019+0000 7f044a7fc640 10 client.4655.objecter ms_dispatch 0x7f0454000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:37:46.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.019+0000 7f044a7fc640 3 client.4655.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:37:46.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.019+0000 7f044a7fc640 3 client.4655.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:37:46.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.019+0000 7f044a7fc640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-25T18:37:46.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.019+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f044c0506b0 con 0x7f04541430c0 2024-03-25T18:37:46.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.019+0000 7f044a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:46.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.019+0000 7f044a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:46.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.274+0000 7f045bf8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:46.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.274+0000 7f045bf8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:46.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.274+0000 7f045bf8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:37:46.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.274+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04540b6fb0 con 0x7f04541430c0 2024-03-25T18:37:46.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.358+0000 7f044a7fc640 1 -- 172.21.15.55:0/2709033034 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f04540b6fb0 con 0x7f04541430c0 2024-03-25T18:37:46.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.358+0000 7f044a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:46.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.358+0000 7f044a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:46.360 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:37:46.360 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:46.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.360+0000 7f045bf8d640 10 librados: watch_flush enter 2024-03-25T18:37:46.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.360+0000 7f045bf8d640 10 librados: watch_flush exit 2024-03-25T18:37:46.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.360+0000 7f045bf8d640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-25T18:37:46.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.360+0000 7f045bf8d640 10 client.4655.objecter successfully canceled tick 2024-03-25T18:37:46.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.361+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 >> v1:172.21.15.55:6800/286536064 conn(0x7f043806dea0 legacy=0x7f0438070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:46.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.361+0000 7f045bf8d640 10 monclient: shutdown 2024-03-25T18:37:46.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.361+0000 7f045bf8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:46.363 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.361+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 >> v1:172.21.15.55:6790/0 conn(0x7f04541430c0 legacy=0x7f045412a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:46.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.361+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 shutdown_connections 2024-03-25T18:37:46.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.361+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 >> 172.21.15.55:0/2709033034 conn(0x7f04540b7ea0 msgr2=0x7f0454074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:46.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.361+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 shutdown_connections 2024-03-25T18:37:46.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.361+0000 7f045bf8d640 1 -- 172.21.15.55:0/2709033034 wait complete. 2024-03-25T18:37:46.364 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:46.361+0000 7f045bf8d640 1 librados: shutdown 2024-03-25T18:37:46.374 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:53.181 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.296+0000 7f11fea8f640 10 monclient: get_monmap_and_config 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.296+0000 7f11fea8f640 10 monclient: build_initial_monmap 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.296+0000 7f11fea8f640 10 monclient: monmap: 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:37:53.297083+0000 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:37:53.297083+0000 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:53.297 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:53.298 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:37:53.298 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:37:53.298 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:53.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 1 Processor -- start 2024-03-25T18:37:53.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 1 -- start start 2024-03-25T18:37:53.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 10 monclient: init 2024-03-25T18:37:53.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:53.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:37:53.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 10 monclient(hunting): picked mon.noname-b con 0x7f11f80a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:53.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 10 monclient(hunting): picked mon.noname-c con 0x7f11f813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:53.299 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 10 monclient(hunting): picked mon.noname-a con 0x7f11f81430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:53.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f8142520 con 0x7f11f81430c0 2024-03-25T18:37:53.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f80b28a0 con 0x7f11f813f710 2024-03-25T18:37:53.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f80b2a30 con 0x7f11f80a98f0 2024-03-25T18:37:53.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:53.300 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.297+0000 7f11fea8f640 10 monclient(hunting): authenticate will time out at 7877.555221s 2024-03-25T18:37:53.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.303+0000 7f11fc82b640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f11f80a98f0 0x7f11f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:40604/0 (socket says 172.21.15.55:40604) 2024-03-25T18:37:53.304 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.303+0000 7f11fc82b640 1 -- 172.21.15.55:0/109170653 learned_addr learned my addr 172.21.15.55:0/109170653 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:53.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.309+0000 7f11f77fe640 1 -- 172.21.15.55:0/109170653 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923611731 0 0) 0x7f11f80b2a30 con 0x7f11f80a98f0 2024-03-25T18:37:53.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.309+0000 7f11f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:37:53.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.309+0000 7f11f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:53.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.309+0000 7f11f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:53.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.309+0000 7f11f77fe640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:37:53.310 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.309+0000 7f11f77fe640 1 -- 172.21.15.55:0/109170653 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e8003520 con 0x7f11f80a98f0 2024-03-25T18:37:53.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 1 -- 172.21.15.55:0/109170653 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 910919788 0 0) 0x7f11e8003520 con 0x7f11f80a98f0 2024-03-25T18:37:53.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 1 -- 172.21.15.55:0/109170653 >> v1:172.21.15.133:6789/0 conn(0x7f11f813f710 legacy=0x7f11f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:37:53.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 1 -- 172.21.15.55:0/109170653 >> v1:172.21.15.55:6789/0 conn(0x7f11f81430c0 legacy=0x7f11f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:37:53.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:37:53.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 1 monclient: found mon.noname-b 2024-03-25T18:37:53.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:53.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.55:6790/0 2024-03-25T18:37:53.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 1 -- 172.21.15.55:0/109170653 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f80b44c0 con 0x7f11f80a98f0 2024-03-25T18:37:53.318 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 10 monclient: _finish_auth 0 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 1 -- 172.21.15.55:0/109170653 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f11ec001690 con 0x7f11f80a98f0 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11f77fe640 10 monclient: dump: 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:53.319 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:53.320 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:53.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11fea8f640 5 monclient: authenticate success, global_id 4658 2024-03-25T18:37:53.320 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.317+0000 7f11fea8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:53.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.329+0000 7f11f77fe640 1 -- 172.21.15.55:0/109170653 <== mon.2 v1:172.21.15.55:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11ec001860 con 0x7f11f80a98f0 2024-03-25T18:37:53.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.329+0000 7f11f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:53.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.329+0000 7f11f77fe640 1 -- 172.21.15.55:0/109170653 <== mon.2 v1:172.21.15.55:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f11ec001cb0 con 0x7f11f80a98f0 2024-03-25T18:37:53.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.329+0000 7f11f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.329+0000 7f11f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.329+0000 7f11f77fe640 10 monclient: dump: 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.329+0000 7f11fea8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:53.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.329+0000 7f11fea8f640 4 set_mon_vals no callback set 2024-03-25T18:37:53.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.330+0000 7f11fea8f640 10 monclient: shutdown 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.330+0000 7f11fea8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.330+0000 7f11fea8f640 1 -- 172.21.15.55:0/109170653 >> v1:172.21.15.55:6790/0 conn(0x7f11f80a98f0 legacy=0x7f11f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.330+0000 7f11fea8f640 1 -- 172.21.15.55:0/109170653 shutdown_connections 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.330+0000 7f11fea8f640 1 -- 172.21.15.55:0/109170653 >> 172.21.15.55:0/109170653 conn(0x7f11f80b7ea0 msgr2=0x7f11f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.330+0000 7f11fea8f640 1 -- 172.21.15.55:0/109170653 shutdown_connections 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.330+0000 7f11fea8f640 1 -- 172.21.15.55:0/109170653 wait complete. 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.331+0000 7f11fea8f640 10 monclient: build_initial_monmap 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.331+0000 7f11fea8f640 10 monclient: monmap: 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:37:53.333 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.331+0000 7f11fea8f640 1 librados: starting msgr at 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.331+0000 7f11fea8f640 1 librados: starting objecter 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.331+0000 7f11fea8f640 1 Processor -- start 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.331+0000 7f11fea8f640 1 -- start start 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.331+0000 7f11fea8f640 1 librados: setting wanted keys 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.331+0000 7f11fea8f640 1 librados: calling monclient init 2024-03-25T18:37:53.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.331+0000 7f11fea8f640 10 monclient: init 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.332+0000 7f11fea8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.332+0000 7f11fea8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.332+0000 7f11fea8f640 10 monclient(hunting): picked mon.noname-a con 0x7f11f80a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.332+0000 7f11fea8f640 10 monclient(hunting): picked mon.noname-b con 0x7f11f813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.332+0000 7f11fea8f640 10 monclient(hunting): picked mon.noname-c con 0x7f11f81430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.332+0000 7f11fea8f640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f8134de0 con 0x7f11f80a98f0 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.333+0000 7f11fea8f640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f8138bd0 con 0x7f11f813f710 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.333+0000 7f11fea8f640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f81394e0 con 0x7f11f81430c0 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.333+0000 7f11fea8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:53.336 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.333+0000 7f11fea8f640 10 monclient(hunting): authenticate will time out at 7877.590397s 2024-03-25T18:37:53.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.342+0000 7f11f7fff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f11f813f710 0x7f11f8135270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:36172/0 (socket says 172.21.15.55:36172) 2024-03-25T18:37:53.342 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.342+0000 7f11f7fff640 1 -- 172.21.15.55:0/3508686076 learned_addr learned my addr 172.21.15.55:0/3508686076 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:37:53.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.343+0000 7f11f57fa640 10 client.?.objecter ms_handle_connect 0x7f11f813f710 2024-03-25T18:37:53.343 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.343+0000 7f11f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:53.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.350+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256152442 0 0) 0x7f11f8138bd0 con 0x7f11f813f710 2024-03-25T18:37:53.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.350+0000 7f11f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:37:53.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.350+0000 7f11f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:53.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.350+0000 7f11f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:53.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.350+0000 7f11f57fa640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:37:53.351 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.350+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc004eb0 con 0x7f11f813f710 2024-03-25T18:37:53.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.354+0000 7f11f57fa640 10 client.?.objecter ms_handle_connect 0x7f11f81430c0 2024-03-25T18:37:53.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.354+0000 7f11f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:53.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.355+0000 7f11f57fa640 10 client.?.objecter ms_handle_connect 0x7f11f80a98f0 2024-03-25T18:37:53.355 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.355+0000 7f11f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:53.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.359+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033917956 0 0) 0x7f11f81394e0 con 0x7f11f81430c0 2024-03-25T18:37:53.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.359+0000 7f11f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:37:53.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.359+0000 7f11f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:53.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.359+0000 7f11f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:53.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.359+0000 7f11f57fa640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:37:53.360 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.359+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f8138bd0 con 0x7f11f81430c0 2024-03-25T18:37:53.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.364+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1636481071 0 0) 0x7f11dc004eb0 con 0x7f11f813f710 2024-03-25T18:37:53.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.364+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f81394e0 con 0x7f11f813f710 2024-03-25T18:37:53.365 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.364+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f11e4001960 con 0x7f11f813f710 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.364+0000 7f11f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.365+0000 7f11f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.365+0000 7f11f57fa640 10 monclient(hunting): dump: 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:53.366 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.365+0000 7f11f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.370+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015125113 0 0) 0x7f11f8134de0 con 0x7f11f80a98f0 2024-03-25T18:37:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.370+0000 7f11f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:37:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.370+0000 7f11f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.370+0000 7f11f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.370+0000 7f11f57fa640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:37:53.371 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.370+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc004eb0 con 0x7f11f80a98f0 2024-03-25T18:37:53.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.371+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4180503103 0 0) 0x7f11f8138bd0 con 0x7f11f81430c0 2024-03-25T18:37:53.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.371+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f8134de0 con 0x7f11f81430c0 2024-03-25T18:37:53.372 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.371+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f11f0001a70 con 0x7f11f81430c0 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.371+0000 7f11f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.371+0000 7f11f57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.371+0000 7f11f57fa640 10 monclient(hunting): dump: 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:53.373 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2515861147 0 0) 0x7f11f81394e0 con 0x7f11f813f710 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 >> v1:172.21.15.55:6790/0 conn(0x7f11f81430c0 legacy=0x7f11f8135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 >> v1:172.21.15.55:6789/0 conn(0x7f11f80a98f0 legacy=0x7f11f81384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 1 monclient: found mon.b 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f81397c0 con 0x7f11f813f710 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 10 monclient: _finish_auth 0 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:53.382 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11fea8f640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:37:53.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11fea8f640 10 monclient: _renew_subs 2024-03-25T18:37:53.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11fea8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:53.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.380+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f80b15c0 con 0x7f11f813f710 2024-03-25T18:37:53.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.381+0000 7f11fea8f640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:53.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.381+0000 7f11fea8f640 10 monclient: _renew_subs 2024-03-25T18:37:53.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.381+0000 7f11fea8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:53.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.381+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f81e2e40 con 0x7f11f813f710 2024-03-25T18:37:53.383 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.381+0000 7f11fea8f640 1 librados: init done 2024-03-25T18:37:53.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.381+0000 7f11fea8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:53.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.381+0000 7f11fea8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:53.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.381+0000 7f11fea8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:53.384 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.381+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f80627b0 con 0x7f11f813f710 2024-03-25T18:37:53.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.665+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11e40035f0 con 0x7f11f813f710 2024-03-25T18:37:53.666 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.665+0000 7f11f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.665+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f11e4003940 con 0x7f11f813f710 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.665+0000 7f11f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.665+0000 7f11f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.665+0000 7f11f57fa640 10 monclient: dump: 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:37:53.667 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:37:53.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.665+0000 7f11f6ffd640 4 set_mon_vals no callback set 2024-03-25T18:37:53.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.666+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f11e4003b60 con 0x7f11f813f710 2024-03-25T18:37:53.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.666+0000 7f11fc82b640 10 monclient: get_auth_request con 0x7f11dc06dfb0 auth_method 0 2024-03-25T18:37:53.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.667+0000 7f11f57fa640 10 client.4645.objecter ms_handle_connect 0x7f11dc06dfb0 2024-03-25T18:37:53.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.669+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f11e4054790 con 0x7f11f813f710 2024-03-25T18:37:53.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.669+0000 7f11f57fa640 10 client.4645.objecter ms_dispatch 0x7f11f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:37:53.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.669+0000 7f11f57fa640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:37:53.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.669+0000 7f11f57fa640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:37:53.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.669+0000 7f11f57fa640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T18:37:53.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.675+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11e40502a0 con 0x7f11f813f710 2024-03-25T18:37:53.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.675+0000 7f11f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:53.675 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.675+0000 7f11f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:53.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.924+0000 7f11fea8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:53.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.924+0000 7f11fea8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:53.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.924+0000 7f11fea8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:37:53.925 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.924+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11f80b71a0 con 0x7f11f813f710 2024-03-25T18:37:53.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.952+0000 7f11f57fa640 1 -- 172.21.15.55:0/3508686076 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f11e40521d0 con 0x7f11f813f710 2024-03-25T18:37:53.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.952+0000 7f11f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:53.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.952+0000 7f11f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:53.953 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:37:53.953 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:53.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.955+0000 7f11fea8f640 10 librados: watch_flush enter 2024-03-25T18:37:53.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.955+0000 7f11fea8f640 10 librados: watch_flush exit 2024-03-25T18:37:53.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.955+0000 7f11fea8f640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T18:37:53.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.955+0000 7f11fea8f640 10 client.4645.objecter successfully canceled tick 2024-03-25T18:37:53.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.955+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 >> v1:172.21.15.55:6800/286536064 conn(0x7f11dc06dfb0 legacy=0x7f11dc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.955+0000 7f11fea8f640 10 monclient: shutdown 2024-03-25T18:37:53.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.955+0000 7f11fea8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:53.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.955+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 >> v1:172.21.15.133:6789/0 conn(0x7f11f813f710 legacy=0x7f11f8135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:53.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.956+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 shutdown_connections 2024-03-25T18:37:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.956+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 >> 172.21.15.55:0/3508686076 conn(0x7f11f80b7ea0 msgr2=0x7f11f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.956+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 shutdown_connections 2024-03-25T18:37:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.956+0000 7f11fea8f640 1 -- 172.21.15.55:0/3508686076 wait complete. 2024-03-25T18:37:53.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:37:53.956+0000 7f11fea8f640 1 librados: shutdown 2024-03-25T18:37:53.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:56.620 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:06.790 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:56.632 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:06.802 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:56.655 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:06.825 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:56.677 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:06.814 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:56.685 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:06.823 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:56.713 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:10:06.883 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:56.723 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:10:06.893 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:56.724 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:06.861 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:56.762 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:10:06.932 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:00.776 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:00.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.892+0000 7f6a8a2e6640 10 monclient: get_monmap_and_config 2024-03-25T18:38:00.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.892+0000 7f6a8a2e6640 10 monclient: build_initial_monmap 2024-03-25T18:38:00.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.892+0000 7f6a8a2e6640 10 monclient: monmap: 2024-03-25T18:38:00.893 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:00.894 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:00.894 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:38:00.893426+0000 2024-03-25T18:38:00.894 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:38:00.893426+0000 2024-03-25T18:38:00.894 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:00.894 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:00.894 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:00.894 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:38:00.894 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:38:00.894 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.893+0000 7f6a8a2e6640 1 Processor -- start 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.893+0000 7f6a8a2e6640 1 -- start start 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.893+0000 7f6a8a2e6640 10 monclient: init 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.894+0000 7f6a8a2e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.894+0000 7f6a8a2e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.894+0000 7f6a8a2e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6a840a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.894+0000 7f6a8a2e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6a8413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.894+0000 7f6a8a2e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6a841430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.894+0000 7f6a8a2e6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a84142520 con 0x7f6a840a98f0 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.894+0000 7f6a8a2e6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a840b28a0 con 0x7f6a8413f710 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.894+0000 7f6a8a2e6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a840b2a30 con 0x7f6a841430c0 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.895+0000 7f6a8a2e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:00.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.895+0000 7f6a8a2e6640 10 monclient(hunting): authenticate will time out at 7885.152402s 2024-03-25T18:38:00.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.903+0000 7f6a88883640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f6a841430c0 0x7f6a8414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:58258/0 (socket says 172.21.15.55:58258) 2024-03-25T18:38:00.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.903+0000 7f6a88883640 1 -- 172.21.15.55:0/2388975678 learned_addr learned my addr 172.21.15.55:0/2388975678 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:00.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.918+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843521781 0 0) 0x7f6a84142520 con 0x7f6a840a98f0 2024-03-25T18:38:00.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.918+0000 7f6a82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:38:00.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.918+0000 7f6a82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:00.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.918+0000 7f6a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:00.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.918+0000 7f6a82ffd640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:38:00.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.918+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a6c003520 con 0x7f6a840a98f0 2024-03-25T18:38:00.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.921+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082205331 0 0) 0x7f6a840b2a30 con 0x7f6a841430c0 2024-03-25T18:38:00.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.921+0000 7f6a82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:38:00.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.921+0000 7f6a82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:00.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.921+0000 7f6a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:00.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.921+0000 7f6a82ffd640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:38:00.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.921+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a84142520 con 0x7f6a841430c0 2024-03-25T18:38:00.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.922+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878766699 0 0) 0x7f6a840b28a0 con 0x7f6a8413f710 2024-03-25T18:38:00.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.922+0000 7f6a82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:38:00.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.922+0000 7f6a82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:00.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.922+0000 7f6a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:00.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.922+0000 7f6a82ffd640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:38:00.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.922+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a840b2a30 con 0x7f6a8413f710 2024-03-25T18:38:00.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3537909222 0 0) 0x7f6a6c003520 con 0x7f6a840a98f0 2024-03-25T18:38:00.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 >> v1:172.21.15.55:6790/0 conn(0x7f6a841430c0 legacy=0x7f6a8414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:00.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 >> v1:172.21.15.133:6789/0 conn(0x7f6a8413f710 legacy=0x7f6a841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:00.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:38:00.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 1 monclient: found mon.noname-a 2024-03-25T18:38:00.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a840b44c0 con 0x7f6a840a98f0 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 10 monclient: _finish_auth 0 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6a74001690 con 0x7f6a840a98f0 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a8a2e6640 5 monclient: authenticate success, global_id 4677 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a8a2e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a82ffd640 10 monclient: dump: 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:00.928 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:00.929 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:00.929 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:00.929 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:00.929 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:00.929 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:00.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.926+0000 7f6a8a2e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:00.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.952+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a740017f0 con 0x7f6a840a98f0 2024-03-25T18:38:00.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.952+0000 7f6a82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:00.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.952+0000 7f6a82ffd640 1 -- 172.21.15.55:0/2388975678 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6a74001c00 con 0x7f6a840a98f0 2024-03-25T18:38:00.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.952+0000 7f6a82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:00.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.952+0000 7f6a82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:38:00.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.952+0000 7f6a82ffd640 10 monclient: dump: 2024-03-25T18:38:00.953 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:00.953 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:00.953 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:00.954 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:00.954 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:00.954 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:00.954 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:00.954 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:00.954 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:00.954 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:00.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.952+0000 7f6a8a2e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:00.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.952+0000 7f6a8a2e6640 4 set_mon_vals no callback set 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.953+0000 7f6a8a2e6640 10 monclient: shutdown 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.953+0000 7f6a8a2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.953+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/2388975678 >> v1:172.21.15.55:6789/0 conn(0x7f6a840a98f0 legacy=0x7f6a840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.953+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/2388975678 shutdown_connections 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.953+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/2388975678 >> 172.21.15.55:0/2388975678 conn(0x7f6a840b7ea0 msgr2=0x7f6a840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.953+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/2388975678 shutdown_connections 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.953+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/2388975678 wait complete. 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.954+0000 7f6a8a2e6640 10 monclient: build_initial_monmap 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.954+0000 7f6a8a2e6640 10 monclient: monmap: 2024-03-25T18:38:00.957 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.954+0000 7f6a8a2e6640 1 librados: starting msgr at 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.954+0000 7f6a8a2e6640 1 librados: starting objecter 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 1 Processor -- start 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 1 -- start start 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 1 librados: setting wanted keys 2024-03-25T18:38:00.958 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 1 librados: calling monclient init 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 10 monclient: init 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6a840a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6a8413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6a841430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a841369b0 con 0x7f6a841430c0 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a841366b0 con 0x7f6a8413f710 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.955+0000 7f6a8a2e6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a84133ad0 con 0x7f6a840a98f0 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.956+0000 7f6a8a2e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:00.959 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.956+0000 7f6a8a2e6640 10 monclient(hunting): authenticate will time out at 7885.213369s 2024-03-25T18:38:00.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.964+0000 7f6a83fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f6a840a98f0 0x7f6a841371e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:58292/0 (socket says 172.21.15.55:58292) 2024-03-25T18:38:00.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.964+0000 7f6a83fff640 1 -- 172.21.15.55:0/3488527027 learned_addr learned my addr 172.21.15.55:0/3488527027 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:00.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.965+0000 7f6a80ff9640 10 client.?.objecter ms_handle_connect 0x7f6a840a98f0 2024-03-25T18:38:00.966 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.965+0000 7f6a80ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:00.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.968+0000 7f6a80ff9640 10 client.?.objecter ms_handle_connect 0x7f6a841430c0 2024-03-25T18:38:00.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.968+0000 7f6a80ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:00.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.968+0000 7f6a80ff9640 10 client.?.objecter ms_handle_connect 0x7f6a8413f710 2024-03-25T18:38:00.969 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.968+0000 7f6a80ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:00.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.969+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789460065 0 0) 0x7f6a84133ad0 con 0x7f6a840a98f0 2024-03-25T18:38:00.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.969+0000 7f6a80ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:38:00.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.969+0000 7f6a80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:00.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.969+0000 7f6a80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:00.970 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.969+0000 7f6a80ff9640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:38:00.974 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.969+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a70004eb0 con 0x7f6a840a98f0 2024-03-25T18:38:00.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.976+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3477800414 0 0) 0x7f6a70004eb0 con 0x7f6a840a98f0 2024-03-25T18:38:00.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.976+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a84133ad0 con 0x7f6a840a98f0 2024-03-25T18:38:00.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.976+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6a74001bc0 con 0x7f6a840a98f0 2024-03-25T18:38:00.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.976+0000 7f6a80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:00.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.976+0000 7f6a80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.976+0000 7f6a80ff9640 10 monclient(hunting): dump: 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:00.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.976+0000 7f6a80ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:00.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.980+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135407369 0 0) 0x7f6a841366b0 con 0x7f6a8413f710 2024-03-25T18:38:00.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.980+0000 7f6a80ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:38:00.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.980+0000 7f6a80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:00.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.980+0000 7f6a80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:00.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.980+0000 7f6a80ff9640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:38:00.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.980+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a70004eb0 con 0x7f6a8413f710 2024-03-25T18:38:00.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1307594099 0 0) 0x7f6a84133ad0 con 0x7f6a840a98f0 2024-03-25T18:38:00.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 >> v1:172.21.15.133:6789/0 conn(0x7f6a8413f710 legacy=0x7f6a84132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:00.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 >> v1:172.21.15.55:6789/0 conn(0x7f6a841430c0 legacy=0x7f6a841333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:00.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:38:00.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 1 monclient: found mon.c 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a84136e40 con 0x7f6a840a98f0 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 10 monclient: _finish_auth 0 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a8a2e6640 5 monclient: authenticate success, global_id 4667 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a8a2e6640 10 monclient: _renew_subs 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a8a2e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.989+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a841378f0 con 0x7f6a840a98f0 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.990+0000 7f6a8a2e6640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.990+0000 7f6a8a2e6640 10 monclient: _renew_subs 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.990+0000 7f6a8a2e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.990+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a840b1480 con 0x7f6a840a98f0 2024-03-25T18:38:00.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.990+0000 7f6a8a2e6640 1 librados: init done 2024-03-25T18:38:00.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.991+0000 7f6a8a2e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:00.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.991+0000 7f6a8a2e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:00.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.991+0000 7f6a8a2e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:00.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:00.991+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a84137030 con 0x7f6a840a98f0 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.311+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a740011f0 con 0x7f6a840a98f0 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.311+0000 7f6a80ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.311+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6a74003b00 con 0x7f6a840a98f0 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.311+0000 7f6a80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.311+0000 7f6a80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.311+0000 7f6a80ff9640 10 monclient: dump: 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:01.313 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:01.314 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:01.314 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:01.314 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:01.314 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:01.314 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:01.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.311+0000 7f6a827fc640 4 set_mon_vals no callback set 2024-03-25T18:38:01.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.312+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f6a74003d20 con 0x7f6a840a98f0 2024-03-25T18:38:01.314 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.312+0000 7f6a837fe640 10 monclient: get_auth_request con 0x7f6a7006dc60 auth_method 0 2024-03-25T18:38:01.315 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.313+0000 7f6a80ff9640 10 client.4667.objecter ms_handle_connect 0x7f6a7006dc60 2024-03-25T18:38:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.317+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f6a74054960 con 0x7f6a840a98f0 2024-03-25T18:38:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.317+0000 7f6a80ff9640 10 client.4667.objecter ms_dispatch 0x7f6a84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:38:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.317+0000 7f6a80ff9640 3 client.4667.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:38:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.317+0000 7f6a80ff9640 3 client.4667.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:38:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.318+0000 7f6a80ff9640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-25T18:38:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.318+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a740502a0 con 0x7f6a840a98f0 2024-03-25T18:38:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.318+0000 7f6a80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:01.319 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.318+0000 7f6a80ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:01.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.544+0000 7f6a8a2e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:01.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.544+0000 7f6a8a2e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:01.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.544+0000 7f6a8a2e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:01.545 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.544+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a840627b0 con 0x7f6a840a98f0 2024-03-25T18:38:01.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.581+0000 7f6a80ff9640 1 -- 172.21.15.55:0/3488527027 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f6a740523a0 con 0x7f6a840a98f0 2024-03-25T18:38:01.582 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.581+0000 7f6a80ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:01.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.581+0000 7f6a80ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:01.583 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:38:01.583 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:01.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 10 librados: watch_flush enter 2024-03-25T18:38:01.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 10 librados: watch_flush exit 2024-03-25T18:38:01.585 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 10 client.4667.objecter successfully canceled tick 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 >> v1:172.21.15.55:6800/286536064 conn(0x7f6a7006dc60 legacy=0x7f6a70070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 10 monclient: shutdown 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 >> v1:172.21.15.55:6790/0 conn(0x7f6a840a98f0 legacy=0x7f6a841371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 shutdown_connections 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 >> 172.21.15.55:0/3488527027 conn(0x7f6a840b7ea0 msgr2=0x7f6a84074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 shutdown_connections 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 1 -- 172.21.15.55:0/3488527027 wait complete. 2024-03-25T18:38:01.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:01.584+0000 7f6a8a2e6640 1 librados: shutdown 2024-03-25T18:38:01.597 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:08.405 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:08.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.518+0000 7f0f1a9af640 10 monclient: get_monmap_and_config 2024-03-25T18:38:08.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.518+0000 7f0f1a9af640 10 monclient: build_initial_monmap 2024-03-25T18:38:08.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.518+0000 7f0f1a9af640 10 monclient: monmap: 2024-03-25T18:38:08.519 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:08.519 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:08.519 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:38:08.519036+0000 2024-03-25T18:38:08.519 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:38:08.519036+0000 2024-03-25T18:38:08.519 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:08.519 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:08.520 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:08.520 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:38:08.520 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:38:08.520 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:08.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.518+0000 7f0f1a9af640 1 Processor -- start 2024-03-25T18:38:08.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.518+0000 7f0f1a9af640 1 -- start start 2024-03-25T18:38:08.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.518+0000 7f0f1a9af640 10 monclient: init 2024-03-25T18:38:08.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:08.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:38:08.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 10 monclient(hunting): picked mon.noname-b con 0x7f0f140b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:08.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 10 monclient(hunting): picked mon.noname-c con 0x7f0f140b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:08.521 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 10 monclient(hunting): picked mon.noname-a con 0x7f0f1414be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:08.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f140af940 con 0x7f0f1414be90 2024-03-25T18:38:08.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f140b0a70 con 0x7f0f140b1730 2024-03-25T18:38:08.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f140b0d50 con 0x7f0f140b4d60 2024-03-25T18:38:08.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:08.522 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.519+0000 7f0f1a9af640 10 monclient(hunting): authenticate will time out at 7892.777087s 2024-03-25T18:38:08.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.529+0000 7f0f137fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f0f140b1730 0x7f0f140b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:48782/0 (socket says 172.21.15.55:48782) 2024-03-25T18:38:08.530 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.529+0000 7f0f137fe640 1 -- 172.21.15.55:0/3285451133 learned_addr learned my addr 172.21.15.55:0/3285451133 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:08.542 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.542+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636051321 0 0) 0x7f0f140b0a70 con 0x7f0f140b1730 2024-03-25T18:38:08.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.542+0000 7f0f12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:38:08.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.542+0000 7f0f12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:08.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.542+0000 7f0f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:08.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.542+0000 7f0f12ffd640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:38:08.543 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.542+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0efc003520 con 0x7f0f140b1730 2024-03-25T18:38:08.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.552+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043045656 0 0) 0x7f0f140b0d50 con 0x7f0f140b4d60 2024-03-25T18:38:08.553 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.553+0000 7f0f12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:38:08.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.553+0000 7f0f12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:08.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.553+0000 7f0f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:08.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.553+0000 7f0f12ffd640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:38:08.554 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.553+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f140b0a70 con 0x7f0f140b4d60 2024-03-25T18:38:08.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1678127965 0 0) 0x7f0efc003520 con 0x7f0f140b1730 2024-03-25T18:38:08.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 >> v1:172.21.15.55:6790/0 conn(0x7f0f140b4d60 legacy=0x7f0f1413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.555 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 >> v1:172.21.15.55:6789/0 conn(0x7f0f1414be90 legacy=0x7f0f1414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 1 monclient: found mon.noname-c 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f140b1030 con 0x7f0f140b1730 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 10 monclient: _finish_auth 0 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0f08001690 con 0x7f0f140b1730 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f12ffd640 10 monclient: dump: 2024-03-25T18:38:08.556 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f1a9af640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:38:08.557 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.554+0000 7f0f1a9af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:08.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.557+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f080017f0 con 0x7f0f140b1730 2024-03-25T18:38:08.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.557+0000 7f0f12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:08.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.557+0000 7f0f12ffd640 1 -- 172.21.15.55:0/3285451133 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0f08001c00 con 0x7f0f140b1730 2024-03-25T18:38:08.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.557+0000 7f0f1a9af640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:08.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.557+0000 7f0f1a9af640 4 set_mon_vals no callback set 2024-03-25T18:38:08.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.557+0000 7f0f12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:08.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.557+0000 7f0f12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:08.559 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.557+0000 7f0f12ffd640 10 monclient: dump: 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.558+0000 7f0f1a9af640 10 monclient: shutdown 2024-03-25T18:38:08.560 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.558+0000 7f0f1a9af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:08.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.558+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3285451133 >> v1:172.21.15.133:6789/0 conn(0x7f0f140b1730 legacy=0x7f0f140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.558+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3285451133 shutdown_connections 2024-03-25T18:38:08.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.558+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3285451133 >> 172.21.15.55:0/3285451133 conn(0x7f0f140617a0 msgr2=0x7f0f14061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:08.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.558+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3285451133 shutdown_connections 2024-03-25T18:38:08.561 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.558+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3285451133 wait complete. 2024-03-25T18:38:08.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.559+0000 7f0f1a9af640 10 monclient: build_initial_monmap 2024-03-25T18:38:08.563 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.559+0000 7f0f1a9af640 10 monclient: monmap: 2024-03-25T18:38:08.563 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:08.563 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:08.563 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:38:08.563 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:38:08.563 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:08.563 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.559+0000 7f0f1a9af640 1 librados: starting msgr at 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.559+0000 7f0f1a9af640 1 librados: starting objecter 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.559+0000 7f0f1a9af640 1 Processor -- start 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.559+0000 7f0f1a9af640 1 -- start start 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.559+0000 7f0f1a9af640 1 librados: setting wanted keys 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.559+0000 7f0f1a9af640 1 librados: calling monclient init 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.559+0000 7f0f1a9af640 10 monclient: init 2024-03-25T18:38:08.564 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 10 monclient(hunting): picked mon.noname-b con 0x7f0f140b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 10 monclient(hunting): picked mon.noname-a con 0x7f0f140b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 10 monclient(hunting): picked mon.noname-c con 0x7f0f1414be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f141db590 con 0x7f0f140b4d60 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f14141fc0 con 0x7f0f140b1730 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f14140590 con 0x7f0f1414be90 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.560+0000 7f0f1a9af640 10 monclient(hunting): authenticate will time out at 7892.817966s 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.562+0000 7f0f137fe640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f0f140b4d60 0x7f0f14134e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:47950/0 (socket says 172.21.15.55:47950) 2024-03-25T18:38:08.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.562+0000 7f0f137fe640 1 -- 172.21.15.55:0/3473049521 learned_addr learned my addr 172.21.15.55:0/3473049521 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:08.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.564+0000 7f0f10ff9640 10 client.?.objecter ms_handle_connect 0x7f0f140b4d60 2024-03-25T18:38:08.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.564+0000 7f0f10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:08.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.570+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583187252 0 0) 0x7f0f141db590 con 0x7f0f140b4d60 2024-03-25T18:38:08.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.570+0000 7f0f10ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:38:08.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.570+0000 7f0f10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:08.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.570+0000 7f0f10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:08.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.570+0000 7f0f10ff9640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:38:08.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.570+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f00004eb0 con 0x7f0f140b4d60 2024-03-25T18:38:08.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.582+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2580266816 0 0) 0x7f0f00004eb0 con 0x7f0f140b4d60 2024-03-25T18:38:08.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.582+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f141db590 con 0x7f0f140b4d60 2024-03-25T18:38:08.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.582+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0f08001bc0 con 0x7f0f140b4d60 2024-03-25T18:38:08.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.582+0000 7f0f10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:08.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.582+0000 7f0f10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:38:08.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.582+0000 7f0f10ff9640 10 monclient(hunting): dump: 2024-03-25T18:38:08.583 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:08.583 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:08.584 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:08.584 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:08.584 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:08.584 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:08.584 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:08.584 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:08.584 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:08.584 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:08.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.582+0000 7f0f10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:08.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 10 client.?.objecter ms_handle_connect 0x7f0f140b1730 2024-03-25T18:38:08.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:08.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 10 client.?.objecter ms_handle_connect 0x7f0f1414be90 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 616205560 0 0) 0x7f0f141db590 con 0x7f0f140b4d60 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 >> v1:172.21.15.55:6790/0 conn(0x7f0f1414be90 legacy=0x7f0f141e3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 >> v1:172.21.15.133:6789/0 conn(0x7f0f140b1730 legacy=0x7f0f141dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 1 monclient: found mon.a 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f141d5440 con 0x7f0f140b4d60 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 10 monclient: _finish_auth 0 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:38:08.591 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:08.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f1a9af640 5 monclient: authenticate success, global_id 4686 2024-03-25T18:38:08.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f1a9af640 10 monclient: _renew_subs 2024-03-25T18:38:08.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f1a9af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:38:08.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f14135550 con 0x7f0f140b4d60 2024-03-25T18:38:08.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f1a9af640 10 client.4686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:08.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f1a9af640 10 monclient: _renew_subs 2024-03-25T18:38:08.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f1a9af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:38:08.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f14137c40 con 0x7f0f140b4d60 2024-03-25T18:38:08.592 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.589+0000 7f0f1a9af640 1 librados: init done 2024-03-25T18:38:08.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.590+0000 7f0f1a9af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:08.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.590+0000 7f0f1a9af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:08.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.590+0000 7f0f1a9af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:38:08.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.590+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f14140720 con 0x7f0f140b4d60 2024-03-25T18:38:08.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.596+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f08004a90 con 0x7f0f140b4d60 2024-03-25T18:38:08.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.596+0000 7f0f10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:08.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.596+0000 7f0f127fc640 4 set_mon_vals no callback set 2024-03-25T18:38:08.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.915+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f0f08004ea0 con 0x7f0f140b4d60 2024-03-25T18:38:08.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.915+0000 7f0f10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:08.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.915+0000 7f0f10ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:38:08.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.915+0000 7f0f10ff9640 10 monclient: dump: 2024-03-25T18:38:08.916 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:08.916 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:08.916 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:08.916 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:08.916 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:08.917 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:08.917 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:08.917 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:08.917 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:08.917 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:08.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.915+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f0f08005080 con 0x7f0f140b4d60 2024-03-25T18:38:08.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.916+0000 7f0f13fff640 10 monclient: get_auth_request con 0x7f0f0006dcf0 auth_method 0 2024-03-25T18:38:08.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.916+0000 7f0f10ff9640 10 client.4686.objecter ms_handle_connect 0x7f0f0006dcf0 2024-03-25T18:38:08.922 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.922+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7f0f08054a80 con 0x7f0f140b4d60 2024-03-25T18:38:08.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.922+0000 7f0f10ff9640 10 client.4686.objecter ms_dispatch 0x7f0f14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:38:08.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.922+0000 7f0f10ff9640 3 client.4686.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:38:08.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.922+0000 7f0f10ff9640 3 client.4686.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:38:08.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.922+0000 7f0f10ff9640 20 client.4686.objecter dump_active .. 0 homeless 2024-03-25T18:38:08.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.922+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f08050590 con 0x7f0f140b4d60 2024-03-25T18:38:08.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.922+0000 7f0f10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:08.923 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:08.922+0000 7f0f10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:09.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.155+0000 7f0f1a9af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:09.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.155+0000 7f0f1a9af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:09.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.155+0000 7f0f1a9af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:38:09.157 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.155+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f140627b0 con 0x7f0f140b4d60 2024-03-25T18:38:09.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.178+0000 7f0f10ff9640 1 -- 172.21.15.55:0/3473049521 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3958857152) 0x7f0f080524c0 con 0x7f0f140b4d60 2024-03-25T18:38:09.179 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.178+0000 7f0f10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:09.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.178+0000 7f0f10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:09.180 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:38:09.180 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":10,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:28:17.650585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:28:17.650585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:09.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 10 librados: watch_flush enter 2024-03-25T18:38:09.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 10 librados: watch_flush exit 2024-03-25T18:38:09.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 20 client.4686.objecter shutdown clearing up homeless session... 2024-03-25T18:38:09.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 10 client.4686.objecter successfully canceled tick 2024-03-25T18:38:09.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 >> v1:172.21.15.55:6800/286536064 conn(0x7f0f0006dcf0 legacy=0x7f0f000701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:09.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 10 monclient: shutdown 2024-03-25T18:38:09.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:09.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 >> v1:172.21.15.55:6789/0 conn(0x7f0f140b4d60 legacy=0x7f0f14134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:09.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 shutdown_connections 2024-03-25T18:38:09.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 >> 172.21.15.55:0/3473049521 conn(0x7f0f140617a0 msgr2=0x7f0f1413c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:09.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 shutdown_connections 2024-03-25T18:38:09.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 1 -- 172.21.15.55:0/3473049521 wait complete. 2024-03-25T18:38:09.184 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:09.181+0000 7f0f1a9af640 1 librados: shutdown 2024-03-25T18:38:09.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:16.000 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:16.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.113+0000 7feca91c9640 10 monclient: get_monmap_and_config 2024-03-25T18:38:16.114 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.113+0000 7feca91c9640 10 monclient: build_initial_monmap 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.113+0000 7feca91c9640 10 monclient: monmap: 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:38:16.114329+0000 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:38:16.114329+0000 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 1 Processor -- start 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 1 -- start start 2024-03-25T18:38:16.115 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 10 monclient: init 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 10 monclient(hunting): picked mon.noname-a con 0x7feca40a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 10 monclient(hunting): picked mon.noname-c con 0x7feca413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 10 monclient(hunting): picked mon.noname-b con 0x7feca41430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca4142520 con 0x7feca40a98f0 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca40b28a0 con 0x7feca413f710 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca40b2a30 con 0x7feca41430c0 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:16.117 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.114+0000 7feca91c9640 10 monclient(hunting): authenticate will time out at 7900.372299s 2024-03-25T18:38:16.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.126+0000 7feca259c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7feca413f710 0x7feca41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:53356/0 (socket says 172.21.15.55:53356) 2024-03-25T18:38:16.127 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.126+0000 7feca259c640 1 -- 172.21.15.55:0/105880189 learned_addr learned my addr 172.21.15.55:0/105880189 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:16.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.150+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923538536 0 0) 0x7feca40b28a0 con 0x7feca413f710 2024-03-25T18:38:16.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.150+0000 7feca1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:38:16.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.150+0000 7feca1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:16.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.150+0000 7feca1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:16.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.150+0000 7feca1d9b640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:38:16.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.150+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c003520 con 0x7feca413f710 2024-03-25T18:38:16.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841756652 0 0) 0x7feca40b2a30 con 0x7feca41430c0 2024-03-25T18:38:16.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:38:16.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:16.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:16.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:38:16.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca40b28a0 con 0x7feca41430c0 2024-03-25T18:38:16.152 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021171605 0 0) 0x7feca4142520 con 0x7feca40a98f0 2024-03-25T18:38:16.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:38:16.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:16.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:16.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:38:16.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.151+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca40b2a30 con 0x7feca40a98f0 2024-03-25T18:38:16.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243547781 0 0) 0x7fec8c003520 con 0x7feca413f710 2024-03-25T18:38:16.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 >> v1:172.21.15.55:6790/0 conn(0x7feca41430c0 legacy=0x7feca414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:16.173 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 >> v1:172.21.15.55:6789/0 conn(0x7feca40a98f0 legacy=0x7feca40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca40b44c0 con 0x7feca413f710 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fec98001690 con 0x7feca413f710 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca1d9b640 10 monclient: dump: 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:16.174 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.175 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.175 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:16.175 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:16.175 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:16.175 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:16.175 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:16.175 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:16.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca91c9640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:38:16.175 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.172+0000 7feca91c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.195+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec980017f0 con 0x7feca413f710 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.195+0000 7feca1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.195+0000 7feca1d9b640 1 -- 172.21.15.55:0/105880189 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fec98001c00 con 0x7feca413f710 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.195+0000 7feca1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.195+0000 7feca1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.195+0000 7feca1d9b640 10 monclient: dump: 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.196 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.197 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:16.197 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:16.197 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:16.197 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:16.197 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:16.197 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:16.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.195+0000 7feca91c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:16.197 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.195+0000 7feca91c9640 4 set_mon_vals no callback set 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.196+0000 7feca91c9640 10 monclient: shutdown 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.196+0000 7feca91c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.196+0000 7feca91c9640 1 -- 172.21.15.55:0/105880189 >> v1:172.21.15.133:6789/0 conn(0x7feca413f710 legacy=0x7feca41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.196+0000 7feca91c9640 1 -- 172.21.15.55:0/105880189 shutdown_connections 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.196+0000 7feca91c9640 1 -- 172.21.15.55:0/105880189 >> 172.21.15.55:0/105880189 conn(0x7feca40b7ea0 msgr2=0x7feca40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.196+0000 7feca91c9640 1 -- 172.21.15.55:0/105880189 shutdown_connections 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.196+0000 7feca91c9640 1 -- 172.21.15.55:0/105880189 wait complete. 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.197+0000 7feca91c9640 10 monclient: build_initial_monmap 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.197+0000 7feca91c9640 10 monclient: monmap: 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:16.199 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.197+0000 7feca91c9640 1 librados: starting msgr at 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.197+0000 7feca91c9640 1 librados: starting objecter 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.198+0000 7feca91c9640 1 Processor -- start 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.198+0000 7feca91c9640 1 -- start start 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.198+0000 7feca91c9640 1 librados: setting wanted keys 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.198+0000 7feca91c9640 1 librados: calling monclient init 2024-03-25T18:38:16.200 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.198+0000 7feca91c9640 10 monclient: init 2024-03-25T18:38:16.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.198+0000 7feca91c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:16.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.198+0000 7feca91c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:38:16.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.199+0000 7feca91c9640 10 monclient(hunting): picked mon.noname-a con 0x7feca40a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:16.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.199+0000 7feca91c9640 10 monclient(hunting): picked mon.noname-b con 0x7feca413f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:16.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.199+0000 7feca91c9640 10 monclient(hunting): picked mon.noname-c con 0x7feca41430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:16.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.199+0000 7feca91c9640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca4138a70 con 0x7feca40a98f0 2024-03-25T18:38:16.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.199+0000 7feca91c9640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca40aba50 con 0x7feca413f710 2024-03-25T18:38:16.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.199+0000 7feca91c9640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca41d77a0 con 0x7feca41430c0 2024-03-25T18:38:16.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.199+0000 7feca91c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:16.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.199+0000 7feca91c9640 10 monclient(hunting): authenticate will time out at 7900.456463s 2024-03-25T18:38:16.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.209+0000 7feca259c640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7feca413f710 0x7feca41d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:53414/0 (socket says 172.21.15.55:53414) 2024-03-25T18:38:16.210 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.209+0000 7feca259c640 1 -- 172.21.15.55:0/1696048398 learned_addr learned my addr 172.21.15.55:0/1696048398 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:16.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.210+0000 7fec8b7fe640 10 client.?.objecter ms_handle_connect 0x7feca413f710 2024-03-25T18:38:16.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.210+0000 7fec8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:16.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.210+0000 7fec8b7fe640 10 client.?.objecter ms_handle_connect 0x7feca40a98f0 2024-03-25T18:38:16.211 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.210+0000 7fec8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:16.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.224+0000 7fec8b7fe640 10 client.?.objecter ms_handle_connect 0x7feca41430c0 2024-03-25T18:38:16.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.224+0000 7fec8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.228+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625993805 0 0) 0x7feca4138a70 con 0x7feca40a98f0 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.228+0000 7fec8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.228+0000 7fec8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.228+0000 7fec8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.228+0000 7fec8b7fe640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.228+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec90004eb0 con 0x7feca40a98f0 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.229+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455784730 0 0) 0x7feca40aba50 con 0x7feca413f710 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.229+0000 7fec8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.229+0000 7fec8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.229+0000 7fec8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.229+0000 7fec8b7fe640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:38:16.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.229+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca4138a70 con 0x7feca413f710 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.242+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281471410 0 0) 0x7feca4138a70 con 0x7feca413f710 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.242+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca40aba50 con 0x7feca413f710 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.242+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fec98001b40 con 0x7feca413f710 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.242+0000 7fec8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.242+0000 7fec8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.242+0000 7fec8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:16.243 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:16.244 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:16.244 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:16.244 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.242+0000 7fec8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:16.245 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.245+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012477879 0 0) 0x7feca41d77a0 con 0x7feca41430c0 2024-03-25T18:38:16.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.245+0000 7fec8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:38:16.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.245+0000 7fec8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:16.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.245+0000 7fec8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:16.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.245+0000 7fec8b7fe640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:38:16.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.245+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca4138a70 con 0x7feca41430c0 2024-03-25T18:38:16.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.256+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 688971978 0 0) 0x7feca4138a70 con 0x7feca41430c0 2024-03-25T18:38:16.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.256+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca41d77a0 con 0x7feca41430c0 2024-03-25T18:38:16.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.256+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fec9c001bd0 con 0x7feca41430c0 2024-03-25T18:38:16.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.256+0000 7fec8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.256+0000 7fec8b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.256+0000 7fec8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:16.258 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:16.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.257+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209834564 0 0) 0x7fec90004eb0 con 0x7feca40a98f0 2024-03-25T18:38:16.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.257+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca4138a70 con 0x7feca40a98f0 2024-03-25T18:38:16.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.257+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fec940018f0 con 0x7feca40a98f0 2024-03-25T18:38:16.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.257+0000 7fec8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:16.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.257+0000 7fec8b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.257+0000 7fec8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:16.260 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:16.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2362502408 0 0) 0x7feca40aba50 con 0x7feca413f710 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 >> v1:172.21.15.55:6790/0 conn(0x7feca41430c0 legacy=0x7feca41d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 >> v1:172.21.15.55:6789/0 conn(0x7feca40a98f0 legacy=0x7feca4138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 1 monclient: found mon.b 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca41d83d0 con 0x7feca413f710 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7fec8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7feca91c9640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7feca91c9640 10 monclient: _renew_subs 2024-03-25T18:38:16.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7feca91c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:16.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca41d41e0 con 0x7feca413f710 2024-03-25T18:38:16.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7feca91c9640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:16.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7feca91c9640 10 monclient: _renew_subs 2024-03-25T18:38:16.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7feca91c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:16.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feca41d5110 con 0x7feca413f710 2024-03-25T18:38:16.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.261+0000 7feca91c9640 1 librados: init done 2024-03-25T18:38:16.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.263+0000 7feca91c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:16.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.263+0000 7feca91c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:16.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.263+0000 7feca91c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:16.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.263+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca414f320 con 0x7feca413f710 2024-03-25T18:38:16.565 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.565+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec980011f0 con 0x7feca413f710 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.565+0000 7fec8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.565+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fec98004c30 con 0x7feca413f710 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.565+0000 7fec8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.565+0000 7fec8b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.565+0000 7fec8b7fe640 10 monclient: dump: 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:16.566 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:16.567 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:16.567 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:16.567 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:16.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.565+0000 7feca159a640 4 set_mon_vals no callback set 2024-03-25T18:38:16.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.565+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fec9801ac80 con 0x7feca413f710 2024-03-25T18:38:16.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.566+0000 7feca2d9d640 10 monclient: get_auth_request con 0x7fec9006e0b0 auth_method 0 2024-03-25T18:38:16.567 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.566+0000 7fec8b7fe640 10 client.4663.objecter ms_handle_connect 0x7fec9006e0b0 2024-03-25T18:38:16.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.570+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2145859450 0 0) 0x7fec98054950 con 0x7feca413f710 2024-03-25T18:38:16.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.570+0000 7fec8b7fe640 10 client.4663.objecter ms_dispatch 0x7feca4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:38:16.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.570+0000 7fec8b7fe640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:38:16.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.570+0000 7fec8b7fe640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:38:16.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.570+0000 7fec8b7fe640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T18:38:16.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.571+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec98050290 con 0x7feca413f710 2024-03-25T18:38:16.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.571+0000 7fec8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:16.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.571+0000 7fec8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:16.798 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.797+0000 7feca91c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:16.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.797+0000 7feca91c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:16.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.797+0000 7feca91c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:16.799 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:16.797+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feca41d7930 con 0x7feca413f710 2024-03-25T18:38:16.885 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:27.054 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:16.938 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:27.108 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:16.971 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:27.141 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:16.995 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:10:27.165 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:17.004 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:10:27.175 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:17.010 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:27.147 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:17.015 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.014+0000 7fec8b7fe640 1 -- 172.21.15.55:0/1696048398 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7fec98052390 con 0x7feca413f710 2024-03-25T18:38:17.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.015+0000 7fec8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:17.016 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.015+0000 7fec8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:17.016 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:38:17.016 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:17.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 10 librados: watch_flush enter 2024-03-25T18:38:17.018 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 10 librados: watch_flush exit 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 10 client.4663.objecter successfully canceled tick 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 >> v1:172.21.15.55:6800/286536064 conn(0x7fec9006e0b0 legacy=0x7fec90070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 10 monclient: shutdown 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 >> v1:172.21.15.133:6789/0 conn(0x7feca413f710 legacy=0x7feca41d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 shutdown_connections 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 >> 172.21.15.55:0/1696048398 conn(0x7feca40b7ea0 msgr2=0x7feca413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 shutdown_connections 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 1 -- 172.21.15.55:0/1696048398 wait complete. 2024-03-25T18:38:17.019 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:17.017+0000 7feca91c9640 1 librados: shutdown 2024-03-25T18:38:17.020 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:27.155 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:17.028 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:17.031 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:10:27.202 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:17.050 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:27.188 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:23.835 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:23.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.952+0000 7fac65058640 10 monclient: get_monmap_and_config 2024-03-25T18:38:23.953 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.952+0000 7fac65058640 10 monclient: build_initial_monmap 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.952+0000 7fac65058640 10 monclient: monmap: 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:38:23.953636+0000 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:38:23.953636+0000 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:38:23.954 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:23.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.953+0000 7fac65058640 1 Processor -- start 2024-03-25T18:38:23.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.953+0000 7fac65058640 1 -- start start 2024-03-25T18:38:23.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.953+0000 7fac65058640 10 monclient: init 2024-03-25T18:38:23.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:23.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:38:23.955 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 10 monclient(hunting): picked mon.noname-b con 0x7fac600a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:23.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 10 monclient(hunting): picked mon.noname-a con 0x7fac6013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:23.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 10 monclient(hunting): picked mon.noname-c con 0x7fac601430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:23.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac60142520 con 0x7fac6013f710 2024-03-25T18:38:23.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac600b28a0 con 0x7fac601430c0 2024-03-25T18:38:23.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac600b2a30 con 0x7fac600a98f0 2024-03-25T18:38:23.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:23.956 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.954+0000 7fac65058640 10 monclient(hunting): authenticate will time out at 7908.212120s 2024-03-25T18:38:23.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.960+0000 7fac5f59e640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fac601430c0 0x7fac6014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:52618/0 (socket says 172.21.15.55:52618) 2024-03-25T18:38:23.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.960+0000 7fac5f59e640 1 -- 172.21.15.55:0/382816064 learned_addr learned my addr 172.21.15.55:0/382816064 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:23.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.989+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643156605 0 0) 0x7fac600b2a30 con 0x7fac600a98f0 2024-03-25T18:38:23.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.989+0000 7fac5e59c640 10 monclient(hunting): _negotiate 2024-03-25T18:38:23.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.989+0000 7fac5e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:23.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.989+0000 7fac5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:23.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.989+0000 7fac5e59c640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:38:23.991 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.989+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac4c003520 con 0x7fac600a98f0 2024-03-25T18:38:23.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.991+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183847282 0 0) 0x7fac600b28a0 con 0x7fac601430c0 2024-03-25T18:38:23.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.991+0000 7fac5e59c640 10 monclient(hunting): _negotiate 2024-03-25T18:38:23.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.991+0000 7fac5e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:23.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.991+0000 7fac5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:23.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.991+0000 7fac5e59c640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:38:23.992 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:23.991+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac600b2a30 con 0x7fac601430c0 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3221876837 0 0) 0x7fac600b2a30 con 0x7fac601430c0 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 >> v1:172.21.15.55:6790/0 conn(0x7fac600a98f0 legacy=0x7fac600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 >> v1:172.21.15.55:6789/0 conn(0x7fac6013f710 legacy=0x7fac601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 10 monclient: _finish_hunting 0 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 1 monclient: found mon.noname-c 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac600b44c0 con 0x7fac601430c0 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 10 monclient: _finish_auth 0 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 10 monclient: _check_auth_tickets 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:24.005 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac580019f0 con 0x7fac601430c0 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac5e59c640 10 monclient: dump: 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:24.006 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac65058640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:38:24.007 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.004+0000 7fac65058640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:24.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.025+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac58001b50 con 0x7fac601430c0 2024-03-25T18:38:24.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.025+0000 7fac5e59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:24.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.025+0000 7fac5e59c640 1 -- 172.21.15.55:0/382816064 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac58001f20 con 0x7fac601430c0 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.025+0000 7fac5e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.025+0000 7fac5e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.025+0000 7fac5e59c640 10 monclient: dump: 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.026+0000 7fac65058640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:24.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.026+0000 7fac65058640 4 set_mon_vals no callback set 2024-03-25T18:38:24.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.026+0000 7fac65058640 10 monclient: shutdown 2024-03-25T18:38:24.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.026+0000 7fac65058640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:24.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.026+0000 7fac65058640 1 -- 172.21.15.55:0/382816064 >> v1:172.21.15.133:6789/0 conn(0x7fac601430c0 legacy=0x7fac6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.026+0000 7fac65058640 1 -- 172.21.15.55:0/382816064 shutdown_connections 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.026+0000 7fac65058640 1 -- 172.21.15.55:0/382816064 >> 172.21.15.55:0/382816064 conn(0x7fac600b7ea0 msgr2=0x7fac600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.026+0000 7fac65058640 1 -- 172.21.15.55:0/382816064 shutdown_connections 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.027+0000 7fac65058640 1 -- 172.21.15.55:0/382816064 wait complete. 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.027+0000 7fac65058640 10 monclient: build_initial_monmap 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.027+0000 7fac65058640 10 monclient: monmap: 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:38:24.030 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 1 librados: starting msgr at 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 1 librados: starting objecter 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 1 Processor -- start 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 1 -- start start 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 1 librados: setting wanted keys 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 1 librados: calling monclient init 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 10 monclient: init 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 10 monclient(hunting): picked mon.noname-a con 0x7fac600a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 10 monclient(hunting): picked mon.noname-b con 0x7fac6013f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 10 monclient(hunting): picked mon.noname-c con 0x7fac601430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac600b02b0 con 0x7fac600a98f0 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac600b0500 con 0x7fac6013f710 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac600aea50 con 0x7fac601430c0 2024-03-25T18:38:24.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:24.032 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.028+0000 7fac65058640 10 monclient(hunting): authenticate will time out at 7908.286279s 2024-03-25T18:38:24.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.045+0000 7fac57fff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fac6013f710 0x7fac600aef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:52660/0 (socket says 172.21.15.55:52660) 2024-03-25T18:38:24.046 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.045+0000 7fac57fff640 1 -- 172.21.15.55:0/3965376937 learned_addr learned my addr 172.21.15.55:0/3965376937 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:24.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.046+0000 7fac577fe640 10 client.?.objecter ms_handle_connect 0x7fac600a98f0 2024-03-25T18:38:24.047 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.046+0000 7fac577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:24.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.049+0000 7fac577fe640 10 client.?.objecter ms_handle_connect 0x7fac6013f710 2024-03-25T18:38:24.050 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.049+0000 7fac577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:24.054 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.054+0000 7fac577fe640 10 client.?.objecter ms_handle_connect 0x7fac601430c0 2024-03-25T18:38:24.055 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.054+0000 7fac577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:24.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.068+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569808419 0 0) 0x7fac600b02b0 con 0x7fac600a98f0 2024-03-25T18:38:24.069 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.069+0000 7fac577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:38:24.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.069+0000 7fac577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:24.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.069+0000 7fac577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:24.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.069+0000 7fac577fe640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:38:24.070 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.069+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac40004eb0 con 0x7fac600a98f0 2024-03-25T18:38:24.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.072+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526278110 0 0) 0x7fac600aea50 con 0x7fac601430c0 2024-03-25T18:38:24.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.072+0000 7fac577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:38:24.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.072+0000 7fac577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:24.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.072+0000 7fac577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:24.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.072+0000 7fac577fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:38:24.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.072+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac600b02b0 con 0x7fac601430c0 2024-03-25T18:38:24.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.075+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652037859 0 0) 0x7fac600b0500 con 0x7fac6013f710 2024-03-25T18:38:24.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.075+0000 7fac577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:38:24.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.075+0000 7fac577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:24.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.075+0000 7fac577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:24.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.075+0000 7fac577fe640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:38:24.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.075+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac600aea50 con 0x7fac6013f710 2024-03-25T18:38:24.112 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.112+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2784667648 0 0) 0x7fac600aea50 con 0x7fac6013f710 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.112+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac600b0500 con 0x7fac6013f710 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.112+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac480018f0 con 0x7fac6013f710 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.112+0000 7fac577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.112+0000 7fac577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.112+0000 7fac577fe640 10 monclient(hunting): dump: 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:24.113 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.112+0000 7fac577fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.129+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3731703985 0 0) 0x7fac600b02b0 con 0x7fac601430c0 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.129+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac600aea50 con 0x7fac601430c0 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.129+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac58001e60 con 0x7fac601430c0 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.129+0000 7fac577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.129+0000 7fac577fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.129+0000 7fac577fe640 10 monclient(hunting): dump: 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.130 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:24.131 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:24.131 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:24.131 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:24.131 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:24.131 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:24.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1864601700 0 0) 0x7fac600b0500 con 0x7fac6013f710 2024-03-25T18:38:24.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 >> v1:172.21.15.55:6790/0 conn(0x7fac601430c0 legacy=0x7fac60133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:24.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 >> v1:172.21.15.55:6789/0 conn(0x7fac600a98f0 legacy=0x7fac600ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:24.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:38:24.134 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 1 monclient: found mon.b 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac600af680 con 0x7fac6013f710 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 10 monclient: _finish_auth 0 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac65058640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac65058640 10 monclient: _renew_subs 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac65058640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac60133780 con 0x7fac6013f710 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac65058640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac65058640 10 monclient: _renew_subs 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac65058640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac600aba90 con 0x7fac6013f710 2024-03-25T18:38:24.135 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.133+0000 7fac65058640 1 librados: init done 2024-03-25T18:38:24.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.134+0000 7fac65058640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:24.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.134+0000 7fac65058640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:24.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.134+0000 7fac65058640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:24.136 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.134+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac60133b20 con 0x7fac6013f710 2024-03-25T18:38:24.425 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.424+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac48003610 con 0x7fac6013f710 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.424+0000 7fac577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.424+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac480037c0 con 0x7fac6013f710 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.424+0000 7fac577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.424+0000 7fac577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.424+0000 7fac577fe640 10 monclient: dump: 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:24.426 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:24.427 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:24.427 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:24.427 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:24.427 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:24.427 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:24.427 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.424+0000 7fac5dd9b640 4 set_mon_vals no callback set 2024-03-25T18:38:24.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.425+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fac480039e0 con 0x7fac6013f710 2024-03-25T18:38:24.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.426+0000 7fac5ed9d640 10 monclient: get_auth_request con 0x7fac4006df80 auth_method 0 2024-03-25T18:38:24.428 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.426+0000 7fac577fe640 10 client.4669.objecter ms_handle_connect 0x7fac4006df80 2024-03-25T18:38:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.430+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7fac480545a0 con 0x7fac6013f710 2024-03-25T18:38:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.430+0000 7fac577fe640 10 client.4669.objecter ms_dispatch 0x7fac60000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:38:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.430+0000 7fac577fe640 3 client.4669.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:38:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.430+0000 7fac577fe640 3 client.4669.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:38:24.431 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.430+0000 7fac577fe640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T18:38:24.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.430+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac480500b0 con 0x7fac6013f710 2024-03-25T18:38:24.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.430+0000 7fac577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:24.432 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.430+0000 7fac577fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:24.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.679+0000 7fac65058640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:24.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.679+0000 7fac65058640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:24.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.679+0000 7fac65058640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:24.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.679+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac60074cc0 con 0x7fac6013f710 2024-03-25T18:38:24.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.709+0000 7fac577fe640 1 -- 172.21.15.55:0/3965376937 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7fac48051fe0 con 0x7fac6013f710 2024-03-25T18:38:24.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.709+0000 7fac577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:24.710 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.709+0000 7fac577fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:24.710 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:38:24.711 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:24.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 10 librados: watch_flush enter 2024-03-25T18:38:24.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 10 librados: watch_flush exit 2024-03-25T18:38:24.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T18:38:24.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 10 client.4669.objecter successfully canceled tick 2024-03-25T18:38:24.713 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 >> v1:172.21.15.55:6800/286536064 conn(0x7fac4006df80 legacy=0x7fac40070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:24.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 10 monclient: shutdown 2024-03-25T18:38:24.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:24.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 >> v1:172.21.15.133:6789/0 conn(0x7fac6013f710 legacy=0x7fac600aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:24.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 shutdown_connections 2024-03-25T18:38:24.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 >> 172.21.15.55:0/3965376937 conn(0x7fac600b7ea0 msgr2=0x7fac60074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:24.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 shutdown_connections 2024-03-25T18:38:24.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 1 -- 172.21.15.55:0/3965376937 wait complete. 2024-03-25T18:38:24.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:24.712+0000 7fac65058640 1 librados: shutdown 2024-03-25T18:38:24.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:31.533 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:31.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.648+0000 7f4027c7b640 10 monclient: get_monmap_and_config 2024-03-25T18:38:31.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.648+0000 7f4027c7b640 10 monclient: build_initial_monmap 2024-03-25T18:38:31.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.648+0000 7f4027c7b640 10 monclient: monmap: 2024-03-25T18:38:31.649 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:31.649 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:31.649 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:38:31.649060+0000 2024-03-25T18:38:31.649 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:38:31.649060+0000 2024-03-25T18:38:31.649 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:31.649 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:31.650 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:31.650 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:38:31.650 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:38:31.650 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:31.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.648+0000 7f4027c7b640 1 Processor -- start 2024-03-25T18:38:31.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 1 -- start start 2024-03-25T18:38:31.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 10 monclient: init 2024-03-25T18:38:31.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:31.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:38:31.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f40200a98f0 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:31.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f402013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:31.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f40201430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:31.651 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4020142520 con 0x7f40201430c0 2024-03-25T18:38:31.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40200b28a0 con 0x7f40200a98f0 2024-03-25T18:38:31.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40200b2a30 con 0x7f402013f710 2024-03-25T18:38:31.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:31.652 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.649+0000 7f4027c7b640 10 monclient(hunting): authenticate will time out at 7915.907169s 2024-03-25T18:38:31.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.663+0000 7f4025a17640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f40200a98f0 0x7f40200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:49868/0 (socket says 172.21.15.55:49868) 2024-03-25T18:38:31.663 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.663+0000 7f4025a17640 1 -- 172.21.15.55:0/1731633484 learned_addr learned my addr 172.21.15.55:0/1731633484 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:31.679 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.679+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477160200 0 0) 0x7f40200b2a30 con 0x7f402013f710 2024-03-25T18:38:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.679+0000 7f4024a15640 10 monclient(hunting): _negotiate 2024-03-25T18:38:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.679+0000 7f4024a15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.679+0000 7f4024a15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.679+0000 7f4024a15640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:38:31.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.679+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f400c003520 con 0x7f402013f710 2024-03-25T18:38:31.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.683+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422627154 0 0) 0x7f40200b28a0 con 0x7f40200a98f0 2024-03-25T18:38:31.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.683+0000 7f4024a15640 10 monclient(hunting): _negotiate 2024-03-25T18:38:31.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.683+0000 7f4024a15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:31.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.683+0000 7f4024a15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:31.684 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.683+0000 7f4024a15640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:38:31.685 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.683+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40200b2a30 con 0x7f40200a98f0 2024-03-25T18:38:31.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.700+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243654383 0 0) 0x7f4020142520 con 0x7f40201430c0 2024-03-25T18:38:31.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.700+0000 7f4024a15640 10 monclient(hunting): _negotiate 2024-03-25T18:38:31.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.700+0000 7f4024a15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:31.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.700+0000 7f4024a15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:31.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.700+0000 7f4024a15640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:38:31.701 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.700+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40200b28a0 con 0x7f40201430c0 2024-03-25T18:38:31.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3013538692 0 0) 0x7f40200b2a30 con 0x7f40200a98f0 2024-03-25T18:38:31.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 >> v1:172.21.15.55:6790/0 conn(0x7f402013f710 legacy=0x7f40201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:31.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 >> v1:172.21.15.55:6789/0 conn(0x7f40201430c0 legacy=0x7f402014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:31.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 10 monclient: _finish_hunting 0 2024-03-25T18:38:31.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 1 monclient: found mon.noname-c 2024-03-25T18:38:31.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:31.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:38:31.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40200b44c0 con 0x7f40200a98f0 2024-03-25T18:38:31.703 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 10 monclient: _finish_auth 0 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 10 monclient: _check_auth_tickets 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f401c001670 con 0x7f40200a98f0 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4024a15640 10 monclient: dump: 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:31.704 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:31.705 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:31.705 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:31.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4027c7b640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:38:31.705 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.702+0000 7f4027c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:31.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.710+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f401c0017d0 con 0x7f40200a98f0 2024-03-25T18:38:31.711 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.710+0000 7f4024a15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.710+0000 7f4024a15640 1 -- 172.21.15.55:0/1731633484 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f401c001be0 con 0x7f40200a98f0 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.710+0000 7f4024a15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.710+0000 7f4024a15640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.710+0000 7f4024a15640 10 monclient: dump: 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.710+0000 7f4027c7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:31.712 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.710+0000 7f4027c7b640 4 set_mon_vals no callback set 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.711+0000 7f4027c7b640 10 monclient: shutdown 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.711+0000 7f4027c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.711+0000 7f4027c7b640 1 -- 172.21.15.55:0/1731633484 >> v1:172.21.15.133:6789/0 conn(0x7f40200a98f0 legacy=0x7f40200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.711+0000 7f4027c7b640 1 -- 172.21.15.55:0/1731633484 shutdown_connections 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.711+0000 7f4027c7b640 1 -- 172.21.15.55:0/1731633484 >> 172.21.15.55:0/1731633484 conn(0x7f40200b7ea0 msgr2=0x7f40200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.711+0000 7f4027c7b640 1 -- 172.21.15.55:0/1731633484 shutdown_connections 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.711+0000 7f4027c7b640 1 -- 172.21.15.55:0/1731633484 wait complete. 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.712+0000 7f4027c7b640 10 monclient: build_initial_monmap 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.712+0000 7f4027c7b640 10 monclient: monmap: 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:31.714 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.712+0000 7f4027c7b640 1 librados: starting msgr at 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 1 librados: starting objecter 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 1 Processor -- start 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 1 -- start start 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 1 librados: setting wanted keys 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 1 librados: calling monclient init 2024-03-25T18:38:31.715 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 10 monclient: init 2024-03-25T18:38:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:38:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f40200a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f402013f710 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:31.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f40201430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40201390f0 con 0x7f402013f710 2024-03-25T18:38:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4020138dd0 con 0x7f40201430c0 2024-03-25T18:38:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40201346b0 con 0x7f40200a98f0 2024-03-25T18:38:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:31.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.713+0000 7f4027c7b640 10 monclient(hunting): authenticate will time out at 7915.971286s 2024-03-25T18:38:31.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.723+0000 7f4026218640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f40201430c0 0x7f4020133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:49914/0 (socket says 172.21.15.55:49914) 2024-03-25T18:38:31.724 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.723+0000 7f4026218640 1 -- 172.21.15.55:0/2341438510 learned_addr learned my addr 172.21.15.55:0/2341438510 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:31.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.726+0000 7f40167fc640 10 client.?.objecter ms_handle_connect 0x7f40201430c0 2024-03-25T18:38:31.726 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.726+0000 7f40167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.731+0000 7f40167fc640 10 client.?.objecter ms_handle_connect 0x7f40200a98f0 2024-03-25T18:38:31.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.731+0000 7f40167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:31.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.738+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022882226 0 0) 0x7f4020138dd0 con 0x7f40201430c0 2024-03-25T18:38:31.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.738+0000 7f40167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:38:31.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.738+0000 7f40167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:31.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.738+0000 7f40167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:31.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.738+0000 7f40167fc640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:38:31.739 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.738+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4004004eb0 con 0x7f40201430c0 2024-03-25T18:38:31.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.747+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143469237 0 0) 0x7f40201346b0 con 0x7f40200a98f0 2024-03-25T18:38:31.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.748+0000 7f40167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:38:31.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.748+0000 7f40167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:31.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.748+0000 7f40167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:31.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.748+0000 7f40167fc640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:38:31.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.748+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020138dd0 con 0x7f40200a98f0 2024-03-25T18:38:31.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.748+0000 7f40167fc640 10 client.?.objecter ms_handle_connect 0x7f402013f710 2024-03-25T18:38:31.749 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.748+0000 7f40167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:31.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.750+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3144299926 0 0) 0x7f4004004eb0 con 0x7f40201430c0 2024-03-25T18:38:31.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.750+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40201346b0 con 0x7f40201430c0 2024-03-25T18:38:31.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.750+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4018001d00 con 0x7f40201430c0 2024-03-25T18:38:31.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.750+0000 7f40167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:31.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.750+0000 7f40167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:38:31.751 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.750+0000 7f40167fc640 10 monclient(hunting): dump: 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:31.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.750+0000 7f40167fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:31.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1479704624 0 0) 0x7f40201346b0 con 0x7f40201430c0 2024-03-25T18:38:31.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 >> v1:172.21.15.55:6790/0 conn(0x7f40200a98f0 legacy=0x7f4020137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:31.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 >> v1:172.21.15.55:6789/0 conn(0x7f402013f710 legacy=0x7f40201386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:31.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 10 monclient: _finish_hunting 0 2024-03-25T18:38:31.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 1 monclient: found mon.b 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020134840 con 0x7f40201430c0 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 10 monclient: _finish_auth 0 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 10 monclient: _check_auth_tickets 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f40167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f4027c7b640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f4027c7b640 10 monclient: _renew_subs 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f4027c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4020134a90 con 0x7f40201430c0 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f4027c7b640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f4027c7b640 10 monclient: _renew_subs 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f4027c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40201356f0 con 0x7f40201430c0 2024-03-25T18:38:31.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.761+0000 7f4027c7b640 1 librados: init done 2024-03-25T18:38:31.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.762+0000 7f4027c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:31.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.762+0000 7f4027c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:31.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.762+0000 7f4027c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:31.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:31.762+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40200627b0 con 0x7f40201430c0 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.028+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4018003970 con 0x7f40201430c0 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.028+0000 7f40167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.028+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f4018003d00 con 0x7f40201430c0 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.028+0000 7f40167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.028+0000 7f40167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.028+0000 7f40167fc640 10 monclient: dump: 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:32.029 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:32.030 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:32.030 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:32.030 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:32.030 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:32.030 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:32.030 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:32.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.028+0000 7f4017fff640 4 set_mon_vals no callback set 2024-03-25T18:38:32.030 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.029+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f4018003f20 con 0x7f40201430c0 2024-03-25T18:38:32.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.029+0000 7f4025a17640 10 monclient: get_auth_request con 0x7f400406dd10 auth_method 0 2024-03-25T18:38:32.031 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.030+0000 7f40167fc640 10 client.4675.objecter ms_handle_connect 0x7f400406dd10 2024-03-25T18:38:32.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.034+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7f4018054b70 con 0x7f40201430c0 2024-03-25T18:38:32.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.034+0000 7f40167fc640 10 client.4675.objecter ms_dispatch 0x7f4020000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:38:32.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.034+0000 7f40167fc640 3 client.4675.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:38:32.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.034+0000 7f40167fc640 3 client.4675.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:38:32.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.034+0000 7f40167fc640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T18:38:32.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.035+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4018050680 con 0x7f40201430c0 2024-03-25T18:38:32.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.035+0000 7f40167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:32.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.035+0000 7f40167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:32.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.264+0000 7f4027c7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:32.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.264+0000 7f4027c7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:32.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.264+0000 7f4027c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:32.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.264+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f402013de90 con 0x7f40201430c0 2024-03-25T18:38:32.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.290+0000 7f40167fc640 1 -- 172.21.15.55:0/2341438510 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7f40180525b0 con 0x7f40201430c0 2024-03-25T18:38:32.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.290+0000 7f40167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:32.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.290+0000 7f40167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:32.292 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:38:32.292 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:32.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 10 librados: watch_flush enter 2024-03-25T18:38:32.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 10 librados: watch_flush exit 2024-03-25T18:38:32.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T18:38:32.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 10 client.4675.objecter successfully canceled tick 2024-03-25T18:38:32.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 >> v1:172.21.15.55:6800/286536064 conn(0x7f400406dd10 legacy=0x7f40040701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:32.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 10 monclient: shutdown 2024-03-25T18:38:32.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:32.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 >> v1:172.21.15.133:6789/0 conn(0x7f40201430c0 legacy=0x7f4020133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:32.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 shutdown_connections 2024-03-25T18:38:32.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 >> 172.21.15.55:0/2341438510 conn(0x7f40200b7ea0 msgr2=0x7f4020074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:32.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 shutdown_connections 2024-03-25T18:38:32.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 1 -- 172.21.15.55:0/2341438510 wait complete. 2024-03-25T18:38:32.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:32.293+0000 7f4027c7b640 1 librados: shutdown 2024-03-25T18:38:32.305 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:36.729 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:46.899 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:36.732 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:46.902 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:36.801 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:10:46.971 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:36.844 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:46.981 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:36.846 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:10:47.016 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:36.848 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:46.986 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:36.858 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:10:47.028 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:36.896 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:10:47.066 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:36.896 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:10:47.033 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:39.111 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:39.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.223+0000 7f29e203e640 10 monclient: get_monmap_and_config 2024-03-25T18:38:39.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.223+0000 7f29e203e640 10 monclient: build_initial_monmap 2024-03-25T18:38:39.227 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.223+0000 7f29e203e640 10 monclient: monmap: 2024-03-25T18:38:39.227 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:39.227 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:39.227 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:38:39.225682+0000 2024-03-25T18:38:39.227 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:38:39.225682+0000 2024-03-25T18:38:39.227 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:39.228 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:39.228 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:39.228 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:38:39.228 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:38:39.228 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:39.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.225+0000 7f29e203e640 1 Processor -- start 2024-03-25T18:38:39.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.225+0000 7f29e203e640 1 -- start start 2024-03-25T18:38:39.228 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.225+0000 7f29e203e640 10 monclient: init 2024-03-25T18:38:39.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.225+0000 7f29e203e640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:39.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.225+0000 7f29e203e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:38:39.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.226+0000 7f29e203e640 10 monclient(hunting): picked mon.noname-c con 0x7f29dc0b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:39.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.226+0000 7f29e203e640 10 monclient(hunting): picked mon.noname-b con 0x7f29dc0b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:39.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.226+0000 7f29e203e640 10 monclient(hunting): picked mon.noname-a con 0x7f29dc14be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:39.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.226+0000 7f29e203e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc0af940 con 0x7f29dc14be90 2024-03-25T18:38:39.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.226+0000 7f29e203e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc0b0a70 con 0x7f29dc0b4d60 2024-03-25T18:38:39.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.226+0000 7f29e203e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc0b0d50 con 0x7f29dc0b1730 2024-03-25T18:38:39.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.226+0000 7f29e203e640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:39.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.226+0000 7f29e203e640 10 monclient(hunting): authenticate will time out at 7923.484473s 2024-03-25T18:38:39.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.238+0000 7f29db7fe640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f29dc0b4d60 0x7f29dc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:43394/0 (socket says 172.21.15.55:43394) 2024-03-25T18:38:39.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.238+0000 7f29db7fe640 1 -- 172.21.15.55:0/2693382836 learned_addr learned my addr 172.21.15.55:0/2693382836 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:39.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.247+0000 7f29daffd640 1 -- 172.21.15.55:0/2693382836 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130926087 0 0) 0x7f29dc0b0a70 con 0x7f29dc0b4d60 2024-03-25T18:38:39.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.247+0000 7f29daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:38:39.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.247+0000 7f29daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:39.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.247+0000 7f29daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:39.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.247+0000 7f29daffd640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:38:39.249 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.247+0000 7f29daffd640 1 -- 172.21.15.55:0/2693382836 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29bc003520 con 0x7f29dc0b4d60 2024-03-25T18:38:39.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 1 -- 172.21.15.55:0/2693382836 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2634417958 0 0) 0x7f29bc003520 con 0x7f29dc0b4d60 2024-03-25T18:38:39.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 1 -- 172.21.15.55:0/2693382836 >> v1:172.21.15.55:6790/0 conn(0x7f29dc0b1730 legacy=0x7f29dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:39.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 1 -- 172.21.15.55:0/2693382836 >> v1:172.21.15.55:6789/0 conn(0x7f29dc14be90 legacy=0x7f29dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:39.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 10 monclient: _finish_hunting 0 2024-03-25T18:38:39.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 1 monclient: found mon.noname-c 2024-03-25T18:38:39.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:39.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:38:39.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 1 -- 172.21.15.55:0/2693382836 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc0b1030 con 0x7f29dc0b4d60 2024-03-25T18:38:39.262 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 10 monclient: _finish_auth 0 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 10 monclient: _check_auth_tickets 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 1 -- 172.21.15.55:0/2693382836 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f29cc001690 con 0x7f29dc0b4d60 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29daffd640 10 monclient: dump: 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:39.263 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:39.264 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:39.264 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:39.264 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:39.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29e203e640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:38:39.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.260+0000 7f29e203e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.269+0000 7f29daffd640 1 -- 172.21.15.55:0/2693382836 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29cc0017f0 con 0x7f29dc0b4d60 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.269+0000 7f29daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.269+0000 7f29daffd640 1 -- 172.21.15.55:0/2693382836 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f29cc001c00 con 0x7f29dc0b4d60 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.269+0000 7f29daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.269+0000 7f29daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.269+0000 7f29daffd640 10 monclient: dump: 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.271 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:39.272 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:39.272 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:39.272 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:39.272 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:39.272 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:39.272 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.269+0000 7f29e203e640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:39.272 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.269+0000 7f29e203e640 4 set_mon_vals no callback set 2024-03-25T18:38:39.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.270+0000 7f29e203e640 10 monclient: shutdown 2024-03-25T18:38:39.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.270+0000 7f29e203e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:39.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.270+0000 7f29e203e640 1 -- 172.21.15.55:0/2693382836 >> v1:172.21.15.133:6789/0 conn(0x7f29dc0b4d60 legacy=0x7f29dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:39.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.270+0000 7f29e203e640 1 -- 172.21.15.55:0/2693382836 shutdown_connections 2024-03-25T18:38:39.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.270+0000 7f29e203e640 1 -- 172.21.15.55:0/2693382836 >> 172.21.15.55:0/2693382836 conn(0x7f29dc0617a0 msgr2=0x7f29dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:39.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.270+0000 7f29e203e640 1 -- 172.21.15.55:0/2693382836 shutdown_connections 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.270+0000 7f29e203e640 1 -- 172.21.15.55:0/2693382836 wait complete. 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.271+0000 7f29e203e640 10 monclient: build_initial_monmap 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.271+0000 7f29e203e640 10 monclient: monmap: 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.271+0000 7f29e203e640 1 librados: starting msgr at 2024-03-25T18:38:39.275 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.271+0000 7f29e203e640 1 librados: starting objecter 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.271+0000 7f29e203e640 1 Processor -- start 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.271+0000 7f29e203e640 1 -- start start 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.271+0000 7f29e203e640 1 librados: setting wanted keys 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.271+0000 7f29e203e640 1 librados: calling monclient init 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.271+0000 7f29e203e640 10 monclient: init 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 10 monclient(hunting): picked mon.noname-b con 0x7f29dc0b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 10 monclient(hunting): picked mon.noname-c con 0x7f29dc0b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 10 monclient(hunting): picked mon.noname-a con 0x7f29dc14be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc135410 con 0x7f29dc14be90 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc141f40 con 0x7f29dc0b1730 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc1356d0 con 0x7f29dc0b4d60 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:39.276 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.272+0000 7f29e203e640 10 monclient(hunting): authenticate will time out at 7923.530700s 2024-03-25T18:38:39.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.284+0000 7f29d3fff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f29dc0b4d60 0x7f29dc1de4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:47226/0 (socket says 172.21.15.55:47226) 2024-03-25T18:38:39.286 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.284+0000 7f29d3fff640 1 -- 172.21.15.55:0/550341021 learned_addr learned my addr 172.21.15.55:0/550341021 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.286+0000 7f29d8ff9640 10 client.?.objecter ms_handle_connect 0x7f29dc0b1730 2024-03-25T18:38:39.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.286+0000 7f29d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.286+0000 7f29d8ff9640 10 client.?.objecter ms_handle_connect 0x7f29dc0b4d60 2024-03-25T18:38:39.288 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.286+0000 7f29d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.296+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850694142 0 0) 0x7f29dc141f40 con 0x7f29dc0b1730 2024-03-25T18:38:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.296+0000 7f29d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:38:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.296+0000 7f29d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.296+0000 7f29d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.296+0000 7f29d8ff9640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:38:39.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.296+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c0004eb0 con 0x7f29dc0b1730 2024-03-25T18:38:39.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.303+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307322017 0 0) 0x7f29dc1356d0 con 0x7f29dc0b4d60 2024-03-25T18:38:39.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.303+0000 7f29d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:38:39.305 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.303+0000 7f29d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:39.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.303+0000 7f29d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:39.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.303+0000 7f29d8ff9640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:38:39.306 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.303+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29dc141f40 con 0x7f29dc0b4d60 2024-03-25T18:38:39.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.309+0000 7f29d8ff9640 10 client.?.objecter ms_handle_connect 0x7f29dc14be90 2024-03-25T18:38:39.311 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.309+0000 7f29d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:39.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.321+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876089541 0 0) 0x7f29dc135410 con 0x7f29dc14be90 2024-03-25T18:38:39.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.321+0000 7f29d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:38:39.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.321+0000 7f29d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:39.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.321+0000 7f29d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:39.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.321+0000 7f29d8ff9640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:38:39.323 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.321+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29dc1356d0 con 0x7f29dc14be90 2024-03-25T18:38:39.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.323+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 189700330 0 0) 0x7f29dc141f40 con 0x7f29dc0b4d60 2024-03-25T18:38:39.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.323+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29dc135410 con 0x7f29dc0b4d60 2024-03-25T18:38:39.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.323+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f29c4001990 con 0x7f29dc0b4d60 2024-03-25T18:38:39.325 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.323+0000 7f29d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.323+0000 7f29d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.323+0000 7f29d8ff9640 10 monclient(hunting): dump: 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:39.326 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.323+0000 7f29d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:39.329 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.328+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 788560828 0 0) 0x7f29c0004eb0 con 0x7f29dc0b1730 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.328+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29dc141f40 con 0x7f29dc0b1730 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.328+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f29cc001bc0 con 0x7f29dc0b1730 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.328+0000 7f29d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.328+0000 7f29d8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.328+0000 7f29d8ff9640 10 monclient(hunting): dump: 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:39.330 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:39.331 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:39.331 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:39.331 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:39.331 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 90846584 0 0) 0x7f29dc135410 con 0x7f29dc0b4d60 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 >> v1:172.21.15.133:6789/0 conn(0x7f29dc0b1730 legacy=0x7f29dc1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 >> v1:172.21.15.55:6789/0 conn(0x7f29dc14be90 legacy=0x7f29dc142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 1 monclient: found mon.c 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc140d00 con 0x7f29dc0b4d60 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29e203e640 5 monclient: authenticate success, global_id 4697 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29e203e640 10 monclient: _renew_subs 2024-03-25T18:38:39.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29e203e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:39.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29dc1debb0 con 0x7f29dc0b4d60 2024-03-25T18:38:39.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29e203e640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:39.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29e203e640 10 monclient: _renew_subs 2024-03-25T18:38:39.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29e203e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:39.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29dc142f00 con 0x7f29dc0b4d60 2024-03-25T18:38:39.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.329+0000 7f29e203e640 1 librados: init done 2024-03-25T18:38:39.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.331+0000 7f29e203e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:39.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.331+0000 7f29e203e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:39.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.331+0000 7f29e203e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:39.334 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.331+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29dc140f50 con 0x7f29dc0b4d60 2024-03-25T18:38:39.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.655+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29c4003610 con 0x7f29dc0b4d60 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.655+0000 7f29d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.655+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f29c40039e0 con 0x7f29dc0b4d60 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.655+0000 7f29d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.655+0000 7f29d8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.655+0000 7f29d8ff9640 10 monclient: dump: 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:39.658 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:39.659 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:39.659 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:39.659 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.655+0000 7f29da7fc640 4 set_mon_vals no callback set 2024-03-25T18:38:39.659 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.656+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f29c4003c00 con 0x7f29dc0b4d60 2024-03-25T18:38:39.659 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.657+0000 7f29db7fe640 10 monclient: get_auth_request con 0x7f29c006df70 auth_method 0 2024-03-25T18:38:39.659 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.657+0000 7f29d8ff9640 10 client.4697.objecter ms_handle_connect 0x7f29c006df70 2024-03-25T18:38:39.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.665+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7f29c4054820 con 0x7f29dc0b4d60 2024-03-25T18:38:39.667 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.665+0000 7f29d8ff9640 10 client.4697.objecter ms_dispatch 0x7f29dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:38:39.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.665+0000 7f29d8ff9640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:38:39.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.665+0000 7f29d8ff9640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:38:39.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.665+0000 7f29d8ff9640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-25T18:38:39.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.665+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29c4050330 con 0x7f29dc0b4d60 2024-03-25T18:38:39.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.666+0000 7f29d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:39.668 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.666+0000 7f29d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:39.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.926+0000 7f29e203e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:39.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.926+0000 7f29e203e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:39.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.926+0000 7f29e203e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:39.929 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.926+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29dc141da0 con 0x7f29dc0b4d60 2024-03-25T18:38:39.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.962+0000 7f29d8ff9640 1 -- 172.21.15.55:0/550341021 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7f29dc141da0 con 0x7f29dc0b4d60 2024-03-25T18:38:39.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.962+0000 7f29d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:39.965 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.962+0000 7f29d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:39.965 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:38:39.965 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 10 librados: watch_flush enter 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 10 librados: watch_flush exit 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 10 client.4697.objecter successfully canceled tick 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 >> v1:172.21.15.55:6800/286536064 conn(0x7f29c006df70 legacy=0x7f29c0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 10 monclient: shutdown 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 >> v1:172.21.15.55:6790/0 conn(0x7f29dc0b4d60 legacy=0x7f29dc1de4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 shutdown_connections 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 >> 172.21.15.55:0/550341021 conn(0x7f29dc0617a0 msgr2=0x7f29dc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.964+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 shutdown_connections 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.965+0000 7f29e203e640 1 -- 172.21.15.55:0/550341021 wait complete. 2024-03-25T18:38:39.967 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:39.965+0000 7f29e203e640 1 librados: shutdown 2024-03-25T18:38:39.978 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:46.785 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:46.906 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.904+0000 7fbc137f6640 10 monclient: get_monmap_and_config 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.904+0000 7fbc137f6640 10 monclient: build_initial_monmap 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.904+0000 7fbc137f6640 10 monclient: monmap: 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:38:46.906695+0000 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:38:46.906695+0000 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:38:46.907 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:38:46.908 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:46.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.905+0000 7fbc137f6640 1 Processor -- start 2024-03-25T18:38:46.908 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.905+0000 7fbc137f6640 1 -- start start 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.905+0000 7fbc137f6640 10 monclient: init 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0c0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0c13f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0c1430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c142520 con 0x7fbc0c0a98f0 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c0b28a0 con 0x7fbc0c13f710 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c0b2a30 con 0x7fbc0c1430c0 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:46.909 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.906+0000 7fbc137f6640 10 monclient(hunting): authenticate will time out at 7931.165189s 2024-03-25T18:38:46.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.914+0000 7fbc11d93640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7fbc0c1430c0 0x7fbc0c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:51796/0 (socket says 172.21.15.55:51796) 2024-03-25T18:38:46.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.914+0000 7fbc11d93640 1 -- 172.21.15.55:0/2553628666 learned_addr learned my addr 172.21.15.55:0/2553628666 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:46.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.931+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029500349 0 0) 0x7fbc0c0b2a30 con 0x7fbc0c1430c0 2024-03-25T18:38:46.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.931+0000 7fbc03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:38:46.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.931+0000 7fbc03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:46.933 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.931+0000 7fbc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:46.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.931+0000 7fbc03fff640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:38:46.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.931+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf8003520 con 0x7fbc0c1430c0 2024-03-25T18:38:46.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.933+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983269669 0 0) 0x7fbc0c0b28a0 con 0x7fbc0c13f710 2024-03-25T18:38:46.934 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.933+0000 7fbc03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:38:46.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.933+0000 7fbc03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:46.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.933+0000 7fbc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:46.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.933+0000 7fbc03fff640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:38:46.935 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.933+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c0b2a30 con 0x7fbc0c13f710 2024-03-25T18:38:46.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.934+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041840849 0 0) 0x7fbc0c142520 con 0x7fbc0c0a98f0 2024-03-25T18:38:46.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.934+0000 7fbc03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:38:46.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.934+0000 7fbc03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:46.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.934+0000 7fbc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:46.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.934+0000 7fbc03fff640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:38:46.937 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.934+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c0b28a0 con 0x7fbc0c0a98f0 2024-03-25T18:38:46.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 280320372 0 0) 0x7fbc0c0b28a0 con 0x7fbc0c0a98f0 2024-03-25T18:38:46.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 >> v1:172.21.15.55:6790/0 conn(0x7fbc0c1430c0 legacy=0x7fbc0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:46.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 >> v1:172.21.15.133:6789/0 conn(0x7fbc0c13f710 legacy=0x7fbc0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:46.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 10 monclient: _finish_hunting 0 2024-03-25T18:38:46.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 1 monclient: found mon.noname-a 2024-03-25T18:38:46.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:46.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:38:46.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c0b44c0 con 0x7fbc0c0a98f0 2024-03-25T18:38:46.948 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 10 monclient: _finish_auth 0 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 10 monclient: _check_auth_tickets 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbc080015b0 con 0x7fbc0c0a98f0 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc137f6640 5 monclient: authenticate success, global_id 4713 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc137f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc03fff640 10 monclient: dump: 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:46.949 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:46.950 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:46.950 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:46.950 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:46.950 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:46.950 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:46.950 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:46.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.945+0000 7fbc137f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.975+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc08001710 con 0x7fbc0c0a98f0 2024-03-25T18:38:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.975+0000 7fbc03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.975+0000 7fbc03fff640 1 -- 172.21.15.55:0/2553628666 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbc08001b60 con 0x7fbc0c0a98f0 2024-03-25T18:38:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.975+0000 7fbc137f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.975+0000 7fbc137f6640 4 set_mon_vals no callback set 2024-03-25T18:38:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.975+0000 7fbc03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:46.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.975+0000 7fbc03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:38:46.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.975+0000 7fbc03fff640 10 monclient: dump: 2024-03-25T18:38:46.978 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:46.978 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:46.978 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:46.978 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:46.978 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:46.978 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:46.978 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:46.978 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:46.979 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:46.979 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:46.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.976+0000 7fbc137f6640 10 monclient: shutdown 2024-03-25T18:38:46.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.976+0000 7fbc137f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:46.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.976+0000 7fbc137f6640 1 -- 172.21.15.55:0/2553628666 >> v1:172.21.15.55:6789/0 conn(0x7fbc0c0a98f0 legacy=0x7fbc0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:46.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.976+0000 7fbc137f6640 1 -- 172.21.15.55:0/2553628666 shutdown_connections 2024-03-25T18:38:46.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.976+0000 7fbc137f6640 1 -- 172.21.15.55:0/2553628666 >> 172.21.15.55:0/2553628666 conn(0x7fbc0c0b7ea0 msgr2=0x7fbc0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:46.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.976+0000 7fbc137f6640 1 -- 172.21.15.55:0/2553628666 shutdown_connections 2024-03-25T18:38:46.981 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.976+0000 7fbc137f6640 1 -- 172.21.15.55:0/2553628666 wait complete. 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.977+0000 7fbc137f6640 10 monclient: build_initial_monmap 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.977+0000 7fbc137f6640 10 monclient: monmap: 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:38:46.982 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.977+0000 7fbc137f6640 1 librados: starting msgr at 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.977+0000 7fbc137f6640 1 librados: starting objecter 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.977+0000 7fbc137f6640 1 Processor -- start 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.977+0000 7fbc137f6640 1 -- start start 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.977+0000 7fbc137f6640 1 librados: setting wanted keys 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.977+0000 7fbc137f6640 1 librados: calling monclient init 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.977+0000 7fbc137f6640 10 monclient: init 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0c0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0c13f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:46.983 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0c1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:46.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c136830 con 0x7fbc0c0a98f0 2024-03-25T18:38:46.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c137e40 con 0x7fbc0c1430c0 2024-03-25T18:38:46.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0c138120 con 0x7fbc0c13f710 2024-03-25T18:38:46.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:46.984 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.978+0000 7fbc137f6640 10 monclient(hunting): authenticate will time out at 7931.236800s 2024-03-25T18:38:46.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.983+0000 7fbc11d93640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fbc0c1430c0 0x7fbc0c1e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:48220/0 (socket says 172.21.15.55:48220) 2024-03-25T18:38:46.985 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.983+0000 7fbc11d93640 1 -- 172.21.15.55:0/2922571454 learned_addr learned my addr 172.21.15.55:0/2922571454 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:46.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.988+0000 7fbc01ffb640 10 client.?.objecter ms_handle_connect 0x7fbc0c1430c0 2024-03-25T18:38:46.989 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.988+0000 7fbc01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:46.990 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.989+0000 7fbc01ffb640 10 client.?.objecter ms_handle_connect 0x7fbc0c0a98f0 2024-03-25T18:38:46.993 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.989+0000 7fbc01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:46.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.992+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014086926 0 0) 0x7fbc0c137e40 con 0x7fbc0c1430c0 2024-03-25T18:38:46.994 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.992+0000 7fbc01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:38:46.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.992+0000 7fbc01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:46.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.992+0000 7fbc01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:46.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.992+0000 7fbc01ffb640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:38:46.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.992+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf0004eb0 con 0x7fbc0c1430c0 2024-03-25T18:38:46.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.993+0000 7fbc01ffb640 10 client.?.objecter ms_handle_connect 0x7fbc0c13f710 2024-03-25T18:38:46.995 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:46.993+0000 7fbc01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:47.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.000+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487271878 0 0) 0x7fbc0c138120 con 0x7fbc0c13f710 2024-03-25T18:38:47.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.000+0000 7fbc01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:38:47.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.000+0000 7fbc01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:47.002 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.000+0000 7fbc01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:47.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.000+0000 7fbc01ffb640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:38:47.003 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.000+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c137e40 con 0x7fbc0c13f710 2024-03-25T18:38:47.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.011+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165857952 0 0) 0x7fbc0c136830 con 0x7fbc0c0a98f0 2024-03-25T18:38:47.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.011+0000 7fbc01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:38:47.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.011+0000 7fbc01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:47.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.011+0000 7fbc01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:47.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.011+0000 7fbc01ffb640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:38:47.013 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.011+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0c138120 con 0x7fbc0c0a98f0 2024-03-25T18:38:47.020 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.018+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3451588122 0 0) 0x7fbc0c137e40 con 0x7fbc0c13f710 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.018+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc0c136830 con 0x7fbc0c13f710 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.018+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbbfc001930 con 0x7fbc0c13f710 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.018+0000 7fbc01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.018+0000 7fbc01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.018+0000 7fbc01ffb640 10 monclient(hunting): dump: 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:47.021 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:47.022 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:47.022 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:47.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.018+0000 7fbc01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:47.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.020+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1301162806 0 0) 0x7fbbf0004eb0 con 0x7fbc0c1430c0 2024-03-25T18:38:47.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.020+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc0c137e40 con 0x7fbc0c1430c0 2024-03-25T18:38:47.022 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.020+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbc04001b70 con 0x7fbc0c1430c0 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.020+0000 7fbc01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.020+0000 7fbc01ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.020+0000 7fbc01ffb640 10 monclient(hunting): dump: 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:47.023 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:47.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.024+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462839941 0 0) 0x7fbc0c138120 con 0x7fbc0c0a98f0 2024-03-25T18:38:47.026 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.025+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbf0004eb0 con 0x7fbc0c0a98f0 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.025+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbc08001a20 con 0x7fbc0c0a98f0 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.025+0000 7fbc01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.025+0000 7fbc01ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.025+0000 7fbc01ffb640 10 monclient(hunting): dump: 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:47.027 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:47.028 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:47.034 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4098016791 0 0) 0x7fbc0c136830 con 0x7fbc0c13f710 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 >> v1:172.21.15.133:6789/0 conn(0x7fbc0c1430c0 legacy=0x7fbc0c1e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 >> v1:172.21.15.55:6789/0 conn(0x7fbc0c0a98f0 legacy=0x7fbc0c1db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 1 monclient: found mon.c 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c0abac0 con 0x7fbc0c13f710 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 10 monclient: _finish_auth 0 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc137f6640 5 monclient: authenticate success, global_id 4703 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc137f6640 10 monclient: _renew_subs 2024-03-25T18:38:47.035 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc137f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:47.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0c1db7f0 con 0x7fbc0c13f710 2024-03-25T18:38:47.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc137f6640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:47.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc137f6640 10 monclient: _renew_subs 2024-03-25T18:38:47.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc137f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:47.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0c137a10 con 0x7fbc0c13f710 2024-03-25T18:38:47.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.033+0000 7fbc137f6640 1 librados: init done 2024-03-25T18:38:47.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.034+0000 7fbc137f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:47.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.034+0000 7fbc137f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:47.036 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.034+0000 7fbc137f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:47.037 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.034+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0c0627b0 con 0x7fbc0c13f710 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.074+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbfc003610 con 0x7fbc0c13f710 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.074+0000 7fbc01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.074+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fbbfc003920 con 0x7fbc0c13f710 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.074+0000 7fbc01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.074+0000 7fbc01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.074+0000 7fbc01ffb640 10 monclient: dump: 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:47.076 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:47.077 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:47.077 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:47.077 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:47.077 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:47.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.074+0000 7fbc037fe640 4 set_mon_vals no callback set 2024-03-25T18:38:47.077 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.075+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fbbfc0198f0 con 0x7fbc0c13f710 2024-03-25T18:38:47.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.075+0000 7fbc11592640 10 monclient: get_auth_request con 0x7fbbf006e0f0 auth_method 0 2024-03-25T18:38:47.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.075+0000 7fbc01ffb640 10 client.4703.objecter ms_handle_connect 0x7fbbf006e0f0 2024-03-25T18:38:47.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.078+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7fbbfc053500 con 0x7fbc0c13f710 2024-03-25T18:38:47.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.078+0000 7fbc01ffb640 10 client.4703.objecter ms_dispatch 0x7fbc0c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:38:47.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.078+0000 7fbc01ffb640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:38:47.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.078+0000 7fbc01ffb640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:38:47.080 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.078+0000 7fbc01ffb640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-25T18:38:47.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.388+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbfc04f010 con 0x7fbc0c13f710 2024-03-25T18:38:47.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.388+0000 7fbc01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:47.389 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.388+0000 7fbc01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:47.619 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.618+0000 7fbc137f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:47.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.618+0000 7fbc137f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:47.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.618+0000 7fbc137f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:38:47.620 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.618+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc0c13fdc0 con 0x7fbc0c13f710 2024-03-25T18:38:47.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.644+0000 7fbc01ffb640 1 -- 172.21.15.55:0/2922571454 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7fbbfc050f40 con 0x7fbc0c13f710 2024-03-25T18:38:47.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.644+0000 7fbc01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:47.645 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.644+0000 7fbc01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:47.646 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:38:47.646 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:47.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 10 librados: watch_flush enter 2024-03-25T18:38:47.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 10 librados: watch_flush exit 2024-03-25T18:38:47.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-25T18:38:47.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 10 client.4703.objecter successfully canceled tick 2024-03-25T18:38:47.648 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 >> v1:172.21.15.55:6800/286536064 conn(0x7fbbf006e0f0 legacy=0x7fbbf00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:47.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 10 monclient: shutdown 2024-03-25T18:38:47.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:47.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 >> v1:172.21.15.55:6790/0 conn(0x7fbc0c13f710 legacy=0x7fbc0c138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:47.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 shutdown_connections 2024-03-25T18:38:47.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 >> 172.21.15.55:0/2922571454 conn(0x7fbc0c0b7ea0 msgr2=0x7fbc0c13d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:47.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 shutdown_connections 2024-03-25T18:38:47.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 1 -- 172.21.15.55:0/2922571454 wait complete. 2024-03-25T18:38:47.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:47.647+0000 7fbc137f6640 1 librados: shutdown 2024-03-25T18:38:47.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:54.466 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:38:54.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.585+0000 7f835fb38640 10 monclient: get_monmap_and_config 2024-03-25T18:38:54.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.585+0000 7f835fb38640 10 monclient: build_initial_monmap 2024-03-25T18:38:54.586 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.585+0000 7f835fb38640 10 monclient: monmap: 2024-03-25T18:38:54.586 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:54.587 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:54.587 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:38:54.586446+0000 2024-03-25T18:38:54.587 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:38:54.586446+0000 2024-03-25T18:38:54.587 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:54.587 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:54.587 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:54.587 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:38:54.587 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:38:54.587 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:54.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.586+0000 7f835fb38640 1 Processor -- start 2024-03-25T18:38:54.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.586+0000 7f835fb38640 1 -- start start 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.586+0000 7f835fb38640 10 monclient: init 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 10 monclient(hunting): picked mon.noname-a con 0x7f835806c380 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 10 monclient(hunting): picked mon.noname-b con 0x7f83580b12d0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 10 monclient(hunting): picked mon.noname-c con 0x7f83580aedf0 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83580b16f0 con 0x7f835806c380 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83580adfe0 con 0x7f83580aedf0 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83580add00 con 0x7f83580b12d0 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:54.589 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.587+0000 7f835fb38640 10 monclient(hunting): authenticate will time out at 7938.845128s 2024-03-25T18:38:54.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.589+0000 7f835e0d5640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f83580aedf0 0x7f835814e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:50928/0 (socket says 172.21.15.55:50928) 2024-03-25T18:38:54.590 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.589+0000 7f835e0d5640 1 -- 172.21.15.55:0/4088574826 learned_addr learned my addr 172.21.15.55:0/4088574826 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:54.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.603+0000 7f835c8d2640 1 -- 172.21.15.55:0/4088574826 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990179418 0 0) 0x7f83580adfe0 con 0x7f83580aedf0 2024-03-25T18:38:54.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.603+0000 7f835c8d2640 10 monclient(hunting): _negotiate 2024-03-25T18:38:54.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.603+0000 7f835c8d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:54.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.603+0000 7f835c8d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:54.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.603+0000 7f835c8d2640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:38:54.604 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.603+0000 7f835c8d2640 1 -- 172.21.15.55:0/4088574826 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8344003520 con 0x7f83580aedf0 2024-03-25T18:38:54.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 1 -- 172.21.15.55:0/4088574826 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2842746039 0 0) 0x7f8344003520 con 0x7f83580aedf0 2024-03-25T18:38:54.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 1 -- 172.21.15.55:0/4088574826 >> v1:172.21.15.55:6790/0 conn(0x7f83580b12d0 legacy=0x7f83580ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:54.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 1 -- 172.21.15.55:0/4088574826 >> v1:172.21.15.55:6789/0 conn(0x7f835806c380 legacy=0x7f83580b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:54.614 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 10 monclient: _finish_hunting 0 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 1 monclient: found mon.noname-c 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 1 -- 172.21.15.55:0/4088574826 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83580ae2c0 con 0x7f83580aedf0 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 10 monclient: _finish_auth 0 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 10 monclient: _check_auth_tickets 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 1 -- 172.21.15.55:0/4088574826 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8350001a10 con 0x7f83580aedf0 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835c8d2640 10 monclient: dump: 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:54.615 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:54.616 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:54.616 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:54.616 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:54.616 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:54.616 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:54.616 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:54.616 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:54.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835fb38640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:38:54.616 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.613+0000 7f835fb38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.621+0000 7f835c8d2640 1 -- 172.21.15.55:0/4088574826 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8350001c30 con 0x7f83580aedf0 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.621+0000 7f835c8d2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.621+0000 7f835c8d2640 1 -- 172.21.15.55:0/4088574826 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8350001f40 con 0x7f83580aedf0 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.621+0000 7f835c8d2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.621+0000 7f835c8d2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.621+0000 7f835c8d2640 10 monclient: dump: 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:54.622 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:54.623 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:54.623 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:54.623 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:54.623 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:54.623 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:54.623 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:54.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.621+0000 7f835fb38640 10 monclient: get_monmap_and_config success 2024-03-25T18:38:54.623 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.621+0000 7f835fb38640 4 set_mon_vals no callback set 2024-03-25T18:38:54.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.622+0000 7f835fb38640 10 monclient: shutdown 2024-03-25T18:38:54.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.622+0000 7f835fb38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:54.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.622+0000 7f835fb38640 1 -- 172.21.15.55:0/4088574826 >> v1:172.21.15.133:6789/0 conn(0x7f83580aedf0 legacy=0x7f835814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:54.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.622+0000 7f835fb38640 1 -- 172.21.15.55:0/4088574826 shutdown_connections 2024-03-25T18:38:54.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.622+0000 7f835fb38640 1 -- 172.21.15.55:0/4088574826 >> 172.21.15.55:0/4088574826 conn(0x7f83580b5d10 msgr2=0x7f83580b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:54.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.622+0000 7f835fb38640 1 -- 172.21.15.55:0/4088574826 shutdown_connections 2024-03-25T18:38:54.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.622+0000 7f835fb38640 1 -- 172.21.15.55:0/4088574826 wait complete. 2024-03-25T18:38:54.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.623+0000 7f835fb38640 10 monclient: build_initial_monmap 2024-03-25T18:38:54.625 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.623+0000 7f835fb38640 10 monclient: monmap: 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.623+0000 7f835fb38640 1 librados: starting msgr at 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.623+0000 7f835fb38640 1 librados: starting objecter 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.623+0000 7f835fb38640 1 Processor -- start 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.623+0000 7f835fb38640 1 -- start start 2024-03-25T18:38:54.626 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.623+0000 7f835fb38640 1 librados: setting wanted keys 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.623+0000 7f835fb38640 1 librados: calling monclient init 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.623+0000 7f835fb38640 10 monclient: init 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 10 monclient(hunting): picked mon.noname-a con 0x7f835806c380 addr v1:172.21.15.55:6789/0 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 10 monclient(hunting): picked mon.noname-b con 0x7f83580aedf0 addr v1:172.21.15.133:6789/0 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 10 monclient(hunting): picked mon.noname-c con 0x7f83580b12d0 addr v1:172.21.15.55:6790/0 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835806c8e0 con 0x7f835806c380 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83581c3540 con 0x7f83580aedf0 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83581c43f0 con 0x7f83580b12d0 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:54.627 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.624+0000 7f835fb38640 10 monclient(hunting): authenticate will time out at 7938.881833s 2024-03-25T18:38:54.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.627+0000 7f835d0d3640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f83580aedf0 0x7f83581c5480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:50970/0 (socket says 172.21.15.55:50970) 2024-03-25T18:38:54.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.627+0000 7f835d0d3640 1 -- 172.21.15.55:0/824960316 learned_addr learned my addr 172.21.15.55:0/824960316 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:38:54.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.627+0000 7f834e7fc640 10 client.?.objecter ms_handle_connect 0x7f83580aedf0 2024-03-25T18:38:54.628 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.628+0000 7f834e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:54.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.636+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554543208 0 0) 0x7f83581c3540 con 0x7f83580aedf0 2024-03-25T18:38:54.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.636+0000 7f834e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:38:54.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.636+0000 7f834e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:54.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.636+0000 7f834e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:54.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.636+0000 7f834e7fc640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:38:54.637 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.636+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c004eb0 con 0x7f83580aedf0 2024-03-25T18:38:54.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.641+0000 7f834e7fc640 10 client.?.objecter ms_handle_connect 0x7f83580b12d0 2024-03-25T18:38:54.641 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.641+0000 7f834e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:54.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.643+0000 7f834e7fc640 10 client.?.objecter ms_handle_connect 0x7f835806c380 2024-03-25T18:38:54.643 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.643+0000 7f834e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:38:54.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.656+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 273710087 0 0) 0x7f833c004eb0 con 0x7f83580aedf0 2024-03-25T18:38:54.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.656+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83581c3540 con 0x7f83580aedf0 2024-03-25T18:38:54.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.656+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8348001930 con 0x7f83580aedf0 2024-03-25T18:38:54.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.656+0000 7f834e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:54.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.656+0000 7f834e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:38:54.657 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.656+0000 7f834e7fc640 10 monclient(hunting): dump: 2024-03-25T18:38:54.657 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:54.658 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.656+0000 7f834e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:38:54.660 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.660+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739086376 0 0) 0x7f83581c43f0 con 0x7f83580b12d0 2024-03-25T18:38:54.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.660+0000 7f834e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:38:54.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.660+0000 7f834e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:54.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.660+0000 7f834e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:54.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.660+0000 7f834e7fc640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:38:54.661 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.660+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c004eb0 con 0x7f83580b12d0 2024-03-25T18:38:54.669 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126165838 0 0) 0x7f835806c8e0 con 0x7f835806c380 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83581c43f0 con 0x7f835806c380 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1184979055 0 0) 0x7f83581c3540 con 0x7f83580aedf0 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 >> v1:172.21.15.55:6790/0 conn(0x7f83580b12d0 legacy=0x7f83581e6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 >> v1:172.21.15.55:6789/0 conn(0x7f835806c380 legacy=0x7f83581c2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 1 monclient: found mon.b 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:38:54.670 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83581c2070 con 0x7f83580aedf0 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f834e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f835fb38640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f835fb38640 10 monclient: _renew_subs 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f835fb38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83581c72b0 con 0x7f83580aedf0 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f835fb38640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f835fb38640 10 monclient: _renew_subs 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f835fb38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83580a9650 con 0x7f83580aedf0 2024-03-25T18:38:54.671 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.668+0000 7f835fb38640 1 librados: init done 2024-03-25T18:38:54.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.670+0000 7f835fb38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:54.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.670+0000 7f835fb38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:54.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.670+0000 7f835fb38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:54.672 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.670+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835806d790 con 0x7f83580aedf0 2024-03-25T18:38:54.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.944+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83480035f0 con 0x7f83580aedf0 2024-03-25T18:38:54.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.944+0000 7f834e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:38:54.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.944+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8348003820 con 0x7f83580aedf0 2024-03-25T18:38:54.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.944+0000 7f834e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:38:54.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.944+0000 7f834e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:38:54.945 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.944+0000 7f834e7fc640 10 monclient: dump: 2024-03-25T18:38:54.945 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:38:54.945 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:38:54.946 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:54.946 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:38:54.946 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:38:54.946 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:38:54.946 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:38:54.946 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:38:54.946 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:38:54.946 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:38:54.946 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.944+0000 7f834ffff640 4 set_mon_vals no callback set 2024-03-25T18:38:54.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.945+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f8348003a40 con 0x7f83580aedf0 2024-03-25T18:38:54.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.945+0000 7f835d8d4640 10 monclient: get_auth_request con 0x7f833c06de90 auth_method 0 2024-03-25T18:38:54.947 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.946+0000 7f834e7fc640 10 client.4693.objecter ms_handle_connect 0x7f833c06de90 2024-03-25T18:38:54.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.949+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7f83480545c0 con 0x7f83580aedf0 2024-03-25T18:38:54.950 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.949+0000 7f834e7fc640 10 client.4693.objecter ms_dispatch 0x7f8358000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:38:54.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.949+0000 7f834e7fc640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:38:54.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.949+0000 7f834e7fc640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:38:54.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.949+0000 7f834e7fc640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T18:38:54.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.950+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83480500d0 con 0x7f83580aedf0 2024-03-25T18:38:54.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.950+0000 7f834e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:38:54.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:54.950+0000 7f834e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:38:55.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.181+0000 7f835fb38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:55.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.181+0000 7f835fb38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:55.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.181+0000 7f835fb38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:38:55.183 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.181+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83580627b0 con 0x7f83580aedf0 2024-03-25T18:38:55.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.211+0000 7f834e7fc640 1 -- 172.21.15.55:0/824960316 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7f8348052000 con 0x7f83580aedf0 2024-03-25T18:38:55.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.211+0000 7f834e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:38:55.212 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.211+0000 7f834e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:38:55.212 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:38:55.212 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:38:55.214 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 10 librados: watch_flush enter 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 10 librados: watch_flush exit 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 10 client.4693.objecter successfully canceled tick 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 >> v1:172.21.15.55:6800/286536064 conn(0x7f833c06de90 legacy=0x7f833c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 10 monclient: shutdown 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 >> v1:172.21.15.133:6789/0 conn(0x7f83580aedf0 legacy=0x7f83581c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 shutdown_connections 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 >> 172.21.15.55:0/824960316 conn(0x7f83580b5d10 msgr2=0x7f83580b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 shutdown_connections 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 1 -- 172.21.15.55:0/824960316 wait complete. 2024-03-25T18:38:55.215 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:38:55.213+0000 7f835fb38640 1 librados: shutdown 2024-03-25T18:38:55.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:38:56.785 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:06.955 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:56.841 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:07.011 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:56.883 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:07.053 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:56.903 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:07.040 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:56.906 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:07.043 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:56.919 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:07.089 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:56.927 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:07.064 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:56.928 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:07.098 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:56.952 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:07.122 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:02.032 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.149+0000 7fac7e915640 10 monclient: get_monmap_and_config 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.150+0000 7fac7e915640 10 monclient: build_initial_monmap 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.150+0000 7fac7e915640 10 monclient: monmap: 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:39:02.150872+0000 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:39:02.150872+0000 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:39:02.151 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:39:02.152 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:02.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 1 Processor -- start 2024-03-25T18:39:02.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 1 -- start start 2024-03-25T18:39:02.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 10 monclient: init 2024-03-25T18:39:02.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:02.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:39:02.153 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 10 monclient(hunting): picked mon.noname-b con 0x7fac780b4d60 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:02.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 10 monclient(hunting): picked mon.noname-a con 0x7fac780b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:02.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 10 monclient(hunting): picked mon.noname-c con 0x7fac7814be90 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:02.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780af940 con 0x7fac780b1730 2024-03-25T18:39:02.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780b0a70 con 0x7fac7814be90 2024-03-25T18:39:02.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780b0d50 con 0x7fac780b4d60 2024-03-25T18:39:02.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:02.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.151+0000 7fac7e915640 10 monclient(hunting): authenticate will time out at 7946.409157s 2024-03-25T18:39:02.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.152+0000 7fac7ceb2640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7fac7814be90 0x7fac7814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:43450/0 (socket says 172.21.15.55:43450) 2024-03-25T18:39:02.154 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.152+0000 7fac7ceb2640 1 -- 172.21.15.55:0/3388942701 learned_addr learned my addr 172.21.15.55:0/3388942701 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:02.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.224+0000 7fac76ffd640 1 -- 172.21.15.55:0/3388942701 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406056763 0 0) 0x7fac780b0a70 con 0x7fac7814be90 2024-03-25T18:39:02.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.224+0000 7fac76ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:02.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.224+0000 7fac76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:02.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.224+0000 7fac76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:02.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.224+0000 7fac76ffd640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:39:02.225 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.224+0000 7fac76ffd640 1 -- 172.21.15.55:0/3388942701 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac60003520 con 0x7fac7814be90 2024-03-25T18:39:02.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 1 -- 172.21.15.55:0/3388942701 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 185426596 0 0) 0x7fac60003520 con 0x7fac7814be90 2024-03-25T18:39:02.229 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 1 -- 172.21.15.55:0/3388942701 >> v1:172.21.15.55:6790/0 conn(0x7fac780b4d60 legacy=0x7fac7813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 1 -- 172.21.15.55:0/3388942701 >> v1:172.21.15.55:6789/0 conn(0x7fac780b1730 legacy=0x7fac780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 1 monclient: found mon.noname-c 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 1 -- 172.21.15.55:0/3388942701 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac780b1030 con 0x7fac7814be90 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 10 monclient: _finish_auth 0 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:02.230 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 1 -- 172.21.15.55:0/3388942701 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac70001a10 con 0x7fac7814be90 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac7e915640 5 monclient: authenticate success, global_id 4696 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac7e915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac76ffd640 10 monclient: dump: 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:02.231 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:02.232 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:02.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.228+0000 7fac7e915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.253+0000 7fac76ffd640 1 -- 172.21.15.55:0/3388942701 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac70001b70 con 0x7fac7814be90 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.253+0000 7fac76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.253+0000 7fac76ffd640 1 -- 172.21.15.55:0/3388942701 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac70001f80 con 0x7fac7814be90 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.253+0000 7fac76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.253+0000 7fac76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.253+0000 7fac76ffd640 10 monclient: dump: 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.254 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:02.255 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:02.255 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:02.255 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:02.255 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:02.255 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:02.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.253+0000 7fac7e915640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:02.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.253+0000 7fac7e915640 4 set_mon_vals no callback set 2024-03-25T18:39:02.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.254+0000 7fac7e915640 10 monclient: shutdown 2024-03-25T18:39:02.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.254+0000 7fac7e915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:02.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.254+0000 7fac7e915640 1 -- 172.21.15.55:0/3388942701 >> v1:172.21.15.133:6789/0 conn(0x7fac7814be90 legacy=0x7fac7814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.254+0000 7fac7e915640 1 -- 172.21.15.55:0/3388942701 shutdown_connections 2024-03-25T18:39:02.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.254+0000 7fac7e915640 1 -- 172.21.15.55:0/3388942701 >> 172.21.15.55:0/3388942701 conn(0x7fac780617a0 msgr2=0x7fac78061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:02.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.254+0000 7fac7e915640 1 -- 172.21.15.55:0/3388942701 shutdown_connections 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.254+0000 7fac7e915640 1 -- 172.21.15.55:0/3388942701 wait complete. 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.255+0000 7fac7e915640 10 monclient: build_initial_monmap 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.255+0000 7fac7e915640 10 monclient: monmap: 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:02.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.256+0000 7fac7e915640 1 librados: starting msgr at 2024-03-25T18:39:02.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.256+0000 7fac7e915640 1 librados: starting objecter 2024-03-25T18:39:02.258 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.256+0000 7fac7e915640 1 Processor -- start 2024-03-25T18:39:02.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.256+0000 7fac7e915640 1 -- start start 2024-03-25T18:39:02.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.256+0000 7fac7e915640 1 librados: setting wanted keys 2024-03-25T18:39:02.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.256+0000 7fac7e915640 1 librados: calling monclient init 2024-03-25T18:39:02.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.256+0000 7fac7e915640 10 monclient: init 2024-03-25T18:39:02.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:02.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:39:02.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 10 monclient(hunting): picked mon.noname-c con 0x7fac780b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:02.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 10 monclient(hunting): picked mon.noname-b con 0x7fac780b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:02.259 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 10 monclient(hunting): picked mon.noname-a con 0x7fac7814be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:02.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac781d7c60 con 0x7fac7814be90 2024-03-25T18:39:02.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac78137200 con 0x7fac780b4d60 2024-03-25T18:39:02.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac781d7830 con 0x7fac780b1730 2024-03-25T18:39:02.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:02.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.257+0000 7fac7e915640 10 monclient(hunting): authenticate will time out at 7946.514798s 2024-03-25T18:39:02.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.258+0000 7fac7ceb2640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7fac7814be90 0x7fac78141750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:56340/0 (socket says 172.21.15.55:56340) 2024-03-25T18:39:02.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.258+0000 7fac7ceb2640 1 -- 172.21.15.55:0/2577783332 learned_addr learned my addr 172.21.15.55:0/2577783332 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:02.260 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.259+0000 7fac74ff9640 10 client.?.objecter ms_handle_connect 0x7fac7814be90 2024-03-25T18:39:02.261 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.259+0000 7fac74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:02.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.263+0000 7fac74ff9640 10 client.?.objecter ms_handle_connect 0x7fac780b4d60 2024-03-25T18:39:02.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.263+0000 7fac74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:02.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.269+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706040031 0 0) 0x7fac78137200 con 0x7fac780b4d60 2024-03-25T18:39:02.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.269+0000 7fac74ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:39:02.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.269+0000 7fac74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:02.270 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.269+0000 7fac74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:02.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.269+0000 7fac74ff9640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:39:02.271 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.269+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac64004eb0 con 0x7fac780b4d60 2024-03-25T18:39:02.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.272+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504996468 0 0) 0x7fac781d7c60 con 0x7fac7814be90 2024-03-25T18:39:02.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.272+0000 7fac74ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:39:02.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.272+0000 7fac74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:02.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.272+0000 7fac74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:02.273 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.272+0000 7fac74ff9640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:39:02.274 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.272+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac78137200 con 0x7fac7814be90 2024-03-25T18:39:02.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.278+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3318021669 0 0) 0x7fac64004eb0 con 0x7fac780b4d60 2024-03-25T18:39:02.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.278+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac781d7c60 con 0x7fac780b4d60 2024-03-25T18:39:02.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.278+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac6c001930 con 0x7fac780b4d60 2024-03-25T18:39:02.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.278+0000 7fac74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:02.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.278+0000 7fac74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:02.279 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.278+0000 7fac74ff9640 10 monclient(hunting): dump: 2024-03-25T18:39:02.279 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:02.279 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:02.280 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.280 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.280 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:02.280 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:02.280 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:02.280 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:02.280 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:02.280 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:02.280 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.278+0000 7fac74ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.284+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 572149517 0 0) 0x7fac78137200 con 0x7fac7814be90 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.284+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac64004eb0 con 0x7fac7814be90 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.284+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac70001f40 con 0x7fac7814be90 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.284+0000 7fac74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.284+0000 7fac74ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.284+0000 7fac74ff9640 10 monclient(hunting): dump: 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.285 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.286 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:02.286 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:02.286 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:02.286 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:02.286 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:02.286 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:02.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.286+0000 7fac74ff9640 10 client.?.objecter ms_handle_connect 0x7fac780b1730 2024-03-25T18:39:02.287 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.286+0000 7fac74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:02.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1709006757 0 0) 0x7fac781d7c60 con 0x7fac780b4d60 2024-03-25T18:39:02.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 >> v1:172.21.15.55:6790/0 conn(0x7fac780b1730 legacy=0x7fac78137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 >> v1:172.21.15.55:6789/0 conn(0x7fac7814be90 legacy=0x7fac78141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:39:02.294 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 1 monclient: found mon.b 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78135e80 con 0x7fac780b4d60 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 10 monclient: _finish_auth 0 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac7e915640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac7e915640 10 monclient: _renew_subs 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac7e915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 --> v1:172.21.15.133:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac78136130 con 0x7fac780b4d60 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac7e915640 10 client.4699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac7e915640 10 monclient: _renew_subs 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac7e915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:39:02.295 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 --> v1:172.21.15.133:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac78142040 con 0x7fac780b4d60 2024-03-25T18:39:02.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.293+0000 7fac7e915640 1 librados: init done 2024-03-25T18:39:02.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.294+0000 7fac7e915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:02.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.294+0000 7fac7e915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:02.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.294+0000 7fac7e915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:39:02.296 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.294+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7813cf10 con 0x7fac780b4d60 2024-03-25T18:39:02.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.597+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac6c0035f0 con 0x7fac780b4d60 2024-03-25T18:39:02.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.597+0000 7fac74ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:02.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.597+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7fac6c003980 con 0x7fac780b4d60 2024-03-25T18:39:02.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.597+0000 7fac74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:02.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.597+0000 7fac74ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:02.598 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.597+0000 7fac74ff9640 10 monclient: dump: 2024-03-25T18:39:02.598 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:02.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.597+0000 7fac767fc640 4 set_mon_vals no callback set 2024-03-25T18:39:02.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.597+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7fac6c003ba0 con 0x7fac780b4d60 2024-03-25T18:39:02.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.598+0000 7fac77fff640 10 monclient: get_auth_request con 0x7fac6406de80 auth_method 0 2024-03-25T18:39:02.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.598+0000 7fac74ff9640 10 client.4699.objecter ms_handle_connect 0x7fac6406de80 2024-03-25T18:39:02.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.605+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7fac6c053600 con 0x7fac780b4d60 2024-03-25T18:39:02.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.605+0000 7fac74ff9640 10 client.4699.objecter ms_dispatch 0x7fac78000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:39:02.606 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.605+0000 7fac74ff9640 3 client.4699.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:39:02.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.605+0000 7fac74ff9640 3 client.4699.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:39:02.607 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.606+0000 7fac74ff9640 20 client.4699.objecter dump_active .. 0 homeless 2024-03-25T18:39:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.607+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac6c04f110 con 0x7fac780b4d60 2024-03-25T18:39:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.607+0000 7fac74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:02.608 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.607+0000 7fac74ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:02.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.860+0000 7fac7e915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:02.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.860+0000 7fac7e915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:02.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.860+0000 7fac7e915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.133:6789/0 2024-03-25T18:39:02.861 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.860+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 --> v1:172.21.15.133:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac781d7e10 con 0x7fac780b4d60 2024-03-25T18:39:02.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.881+0000 7fac74ff9640 1 -- 172.21.15.55:0/2577783332 <== mon.1 v1:172.21.15.133:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7fac781d7e10 con 0x7fac780b4d60 2024-03-25T18:39:02.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.881+0000 7fac74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:02.882 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.881+0000 7fac74ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:02.883 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:39:02.883 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:02.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 10 librados: watch_flush enter 2024-03-25T18:39:02.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 10 librados: watch_flush exit 2024-03-25T18:39:02.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 20 client.4699.objecter shutdown clearing up homeless session... 2024-03-25T18:39:02.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 10 client.4699.objecter successfully canceled tick 2024-03-25T18:39:02.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 >> v1:172.21.15.55:6800/286536064 conn(0x7fac6406de80 legacy=0x7fac64070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 10 monclient: shutdown 2024-03-25T18:39:02.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:02.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 >> v1:172.21.15.133:6789/0 conn(0x7fac780b4d60 legacy=0x7fac781d80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 shutdown_connections 2024-03-25T18:39:02.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 >> 172.21.15.55:0/2577783332 conn(0x7fac780617a0 msgr2=0x7fac7813f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:02.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 shutdown_connections 2024-03-25T18:39:02.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 1 -- 172.21.15.55:0/2577783332 wait complete. 2024-03-25T18:39:02.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:02.884+0000 7fac7e915640 1 librados: shutdown 2024-03-25T18:39:02.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:09.704 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:09.809 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.809+0000 7f8512093640 10 monclient: get_monmap_and_config 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.809+0000 7f8512093640 10 monclient: build_initial_monmap 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.809+0000 7f8512093640 10 monclient: monmap: 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:39:09.809962+0000 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:39:09.809962+0000 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:39:09.810 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.809+0000 7f8512093640 1 Processor -- start 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.809+0000 7f8512093640 1 -- start start 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.809+0000 7f8512093640 10 monclient: init 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 10 monclient(hunting): picked mon.noname-a con 0x7f850c0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 10 monclient(hunting): picked mon.noname-b con 0x7f850c13f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 10 monclient(hunting): picked mon.noname-c con 0x7f850c1430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c142520 con 0x7f850c0a98f0 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0b28a0 con 0x7f850c1430c0 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0b2a30 con 0x7f850c13f710 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:09.812 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.810+0000 7f8512093640 10 monclient(hunting): authenticate will time out at 7954.067886s 2024-03-25T18:39:09.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.811+0000 7f850bfff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f850c1430c0 0x7f850c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:42622/0 (socket says 172.21.15.55:42622) 2024-03-25T18:39:09.813 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.811+0000 7f850bfff640 1 -- 172.21.15.55:0/3739121006 learned_addr learned my addr 172.21.15.55:0/3739121006 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:09.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.874+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898384451 0 0) 0x7f850c0b28a0 con 0x7f850c1430c0 2024-03-25T18:39:09.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.874+0000 7f850a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:09.875 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.874+0000 7f850a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:09.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.874+0000 7f850a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:09.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.874+0000 7f850a7fc640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:39:09.876 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.875+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f4003520 con 0x7f850c1430c0 2024-03-25T18:39:09.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.880+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916829554 0 0) 0x7f850c0b2a30 con 0x7f850c13f710 2024-03-25T18:39:09.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.880+0000 7f850a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:09.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.880+0000 7f850a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:09.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.880+0000 7f850a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:09.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.880+0000 7f850a7fc640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:39:09.881 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.880+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f850c0b28a0 con 0x7f850c13f710 2024-03-25T18:39:09.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402728743 0 0) 0x7f84f4003520 con 0x7f850c1430c0 2024-03-25T18:39:09.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 >> v1:172.21.15.55:6790/0 conn(0x7f850c13f710 legacy=0x7f850c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 >> v1:172.21.15.55:6789/0 conn(0x7f850c0a98f0 legacy=0x7f850c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0b44c0 con 0x7f850c1430c0 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8504001a10 con 0x7f850c1430c0 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f8512093640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f8512093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:09.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f850a7fc640 10 monclient: dump: 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:09.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.888+0000 7f8512093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.912+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8504001b70 con 0x7f850c1430c0 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.912+0000 7f850a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.912+0000 7f850a7fc640 1 -- 172.21.15.55:0/3739121006 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8504001f80 con 0x7f850c1430c0 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.912+0000 7f8512093640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.912+0000 7f8512093640 4 set_mon_vals no callback set 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.912+0000 7f850a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.912+0000 7f850a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.912+0000 7f850a7fc640 10 monclient: dump: 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:09.913 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:09.914 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:09.914 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:09.914 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:09.914 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:09.914 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:09.914 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:09.914 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:09.914 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:09.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.913+0000 7f8512093640 10 monclient: shutdown 2024-03-25T18:39:09.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.913+0000 7f8512093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.913+0000 7f8512093640 1 -- 172.21.15.55:0/3739121006 >> v1:172.21.15.133:6789/0 conn(0x7f850c1430c0 legacy=0x7f850c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.913+0000 7f8512093640 1 -- 172.21.15.55:0/3739121006 shutdown_connections 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.913+0000 7f8512093640 1 -- 172.21.15.55:0/3739121006 >> 172.21.15.55:0/3739121006 conn(0x7f850c0b7ea0 msgr2=0x7f850c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.913+0000 7f8512093640 1 -- 172.21.15.55:0/3739121006 shutdown_connections 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.913+0000 7f8512093640 1 -- 172.21.15.55:0/3739121006 wait complete. 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.914+0000 7f8512093640 10 monclient: build_initial_monmap 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.914+0000 7f8512093640 10 monclient: monmap: 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:09.916 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:39:09.917 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:39:09.917 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:09.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.914+0000 7f8512093640 1 librados: starting msgr at 2024-03-25T18:39:09.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.914+0000 7f8512093640 1 librados: starting objecter 2024-03-25T18:39:09.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.914+0000 7f8512093640 1 Processor -- start 2024-03-25T18:39:09.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 1 -- start start 2024-03-25T18:39:09.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 1 librados: setting wanted keys 2024-03-25T18:39:09.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 1 librados: calling monclient init 2024-03-25T18:39:09.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 10 monclient: init 2024-03-25T18:39:09.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:09.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:09.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 10 monclient(hunting): picked mon.noname-a con 0x7f850c0a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:09.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 10 monclient(hunting): picked mon.noname-b con 0x7f850c13f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:09.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 10 monclient(hunting): picked mon.noname-c con 0x7f850c1430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:09.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c1d83a0 con 0x7f850c0a98f0 2024-03-25T18:39:09.918 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0abe50 con 0x7f850c13f710 2024-03-25T18:39:09.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c138f30 con 0x7f850c1430c0 2024-03-25T18:39:09.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:09.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.915+0000 7f8512093640 10 monclient(hunting): authenticate will time out at 7954.173176s 2024-03-25T18:39:09.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.923+0000 7f850bfff640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f850c1430c0 0x7f850c133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:56462/0 (socket says 172.21.15.55:56462) 2024-03-25T18:39:09.924 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.923+0000 7f850bfff640 1 -- 172.21.15.55:0/3001286438 learned_addr learned my addr 172.21.15.55:0/3001286438 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:09.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.928+0000 7f84f3fff640 10 client.?.objecter ms_handle_connect 0x7f850c1430c0 2024-03-25T18:39:09.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.928+0000 7f84f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:09.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.935+0000 7f84f3fff640 10 client.?.objecter ms_handle_connect 0x7f850c13f710 2024-03-25T18:39:09.936 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.935+0000 7f84f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:09.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.938+0000 7f84f3fff640 10 client.?.objecter ms_handle_connect 0x7f850c0a98f0 2024-03-25T18:39:09.939 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.938+0000 7f84f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:09.942 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.942+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782179458 0 0) 0x7f850c138f30 con 0x7f850c1430c0 2024-03-25T18:39:09.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.942+0000 7f84f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:09.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.942+0000 7f84f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:09.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.942+0000 7f84f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:09.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.942+0000 7f84f3fff640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:39:09.943 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.942+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f8004eb0 con 0x7f850c1430c0 2024-03-25T18:39:09.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.950+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058076096 0 0) 0x7f850c0abe50 con 0x7f850c13f710 2024-03-25T18:39:09.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.950+0000 7f84f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:09.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.950+0000 7f84f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:09.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.950+0000 7f84f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:09.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.950+0000 7f84f3fff640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:39:09.951 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.950+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f850c138f30 con 0x7f850c13f710 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.959+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1995127211 0 0) 0x7f84f8004eb0 con 0x7f850c1430c0 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.959+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f850c0abe50 con 0x7f850c1430c0 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.959+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8504001ec0 con 0x7f850c1430c0 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.959+0000 7f84f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.959+0000 7f84f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.959+0000 7f84f3fff640 10 monclient(hunting): dump: 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:09.960 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:09.961 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:09.961 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:09.961 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:09.961 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:09.961 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:09.961 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.959+0000 7f84f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:09.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.963+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286900629 0 0) 0x7f850c1d83a0 con 0x7f850c0a98f0 2024-03-25T18:39:09.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.963+0000 7f84f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:09.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.963+0000 7f84f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:09.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.963+0000 7f84f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:09.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.963+0000 7f84f3fff640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:39:09.964 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.963+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f8004eb0 con 0x7f850c0a98f0 2024-03-25T18:39:09.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2448866730 0 0) 0x7f850c0abe50 con 0x7f850c1430c0 2024-03-25T18:39:09.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 >> v1:172.21.15.133:6789/0 conn(0x7f850c13f710 legacy=0x7f850c138820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:09.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 >> v1:172.21.15.55:6789/0 conn(0x7f850c0a98f0 legacy=0x7f850c1d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:09.976 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 1 monclient: found mon.c 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c1d7490 con 0x7f850c1430c0 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 10 monclient: _finish_auth 0 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f84f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f8512093640 5 monclient: authenticate success, global_id 4718 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f8512093640 10 monclient: _renew_subs 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f8512093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c1d7770 con 0x7f850c1430c0 2024-03-25T18:39:09.977 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f8512093640 10 client.4718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:09.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f8512093640 10 monclient: _renew_subs 2024-03-25T18:39:09.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f8512093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:09.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c1381f0 con 0x7f850c1430c0 2024-03-25T18:39:09.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.975+0000 7f8512093640 1 librados: init done 2024-03-25T18:39:09.978 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.976+0000 7f8512093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:09.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.976+0000 7f8512093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:09.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.976+0000 7f8512093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:09.979 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:09.976+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c0627b0 con 0x7f850c1430c0 2024-03-25T18:39:10.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.288+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8504004e10 con 0x7f850c1430c0 2024-03-25T18:39:10.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.288+0000 7f84f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:10.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.288+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f8504002100 con 0x7f850c1430c0 2024-03-25T18:39:10.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.288+0000 7f84f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:10.289 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.288+0000 7f84f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.288+0000 7f84f3fff640 10 monclient: dump: 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:10.290 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:10.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.288+0000 7f8509ffb640 4 set_mon_vals no callback set 2024-03-25T18:39:10.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.289+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f850401af90 con 0x7f850c1430c0 2024-03-25T18:39:10.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.289+0000 7f850b7fe640 10 monclient: get_auth_request con 0x7f84f806ddb0 auth_method 0 2024-03-25T18:39:10.291 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.290+0000 7f84f3fff640 10 client.4718.objecter ms_handle_connect 0x7f84f806ddb0 2024-03-25T18:39:10.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.297+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7f8504054b20 con 0x7f850c1430c0 2024-03-25T18:39:10.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.297+0000 7f84f3fff640 10 client.4718.objecter ms_dispatch 0x7f850c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:39:10.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.297+0000 7f84f3fff640 3 client.4718.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:39:10.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.297+0000 7f84f3fff640 3 client.4718.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:39:10.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.297+0000 7f84f3fff640 20 client.4718.objecter dump_active .. 0 homeless 2024-03-25T18:39:10.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.297+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8504050460 con 0x7f850c1430c0 2024-03-25T18:39:10.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.297+0000 7f84f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:10.298 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.297+0000 7f84f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:10.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.540+0000 7f8512093640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:10.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.540+0000 7f8512093640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:10.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.540+0000 7f8512093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:10.541 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.540+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f850c069f40 con 0x7f850c1430c0 2024-03-25T18:39:10.568 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.567+0000 7f84f3fff640 1 -- 172.21.15.55:0/3001286438 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7f8504052560 con 0x7f850c1430c0 2024-03-25T18:39:10.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.567+0000 7f84f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:10.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.567+0000 7f84f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:10.569 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:39:10.569 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:10.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 10 librados: watch_flush enter 2024-03-25T18:39:10.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 10 librados: watch_flush exit 2024-03-25T18:39:10.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 20 client.4718.objecter shutdown clearing up homeless session... 2024-03-25T18:39:10.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 10 client.4718.objecter successfully canceled tick 2024-03-25T18:39:10.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 >> v1:172.21.15.55:6800/286536064 conn(0x7f84f806ddb0 legacy=0x7f84f8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:10.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 10 monclient: shutdown 2024-03-25T18:39:10.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:10.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 >> v1:172.21.15.55:6790/0 conn(0x7f850c1430c0 legacy=0x7f850c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:10.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 shutdown_connections 2024-03-25T18:39:10.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 >> 172.21.15.55:0/3001286438 conn(0x7f850c0b7ea0 msgr2=0x7f850c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:10.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 shutdown_connections 2024-03-25T18:39:10.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 1 -- 172.21.15.55:0/3001286438 wait complete. 2024-03-25T18:39:10.572 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:10.570+0000 7f8512093640 1 librados: shutdown 2024-03-25T18:39:10.583 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:16.929 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:27.099 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:16.936 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:27.106 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:16.977 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:27.147 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:16.983 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:27.120 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:16.993 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:27.130 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:17.003 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:27.173 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:17.006 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:27.177 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:17.020 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:27.157 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:17.039 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:27.209 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:17.390 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:17.508 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.508+0000 7f22f4923640 10 monclient: get_monmap_and_config 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.508+0000 7f22f4923640 10 monclient: build_initial_monmap 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.508+0000 7f22f4923640 10 monclient: monmap: 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:39:17.508934+0000 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:39:17.508934+0000 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:17.509 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:39:17.510 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:39:17.510 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.509+0000 7f22f4923640 1 Processor -- start 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.509+0000 7f22f4923640 1 -- start start 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.509+0000 7f22f4923640 10 monclient: init 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 10 monclient(hunting): picked mon.noname-a con 0x7f22f00a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 10 monclient(hunting): picked mon.noname-c con 0x7f22f013f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 10 monclient(hunting): picked mon.noname-b con 0x7f22f01430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f0142520 con 0x7f22f00a98f0 2024-03-25T18:39:17.511 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f00b28a0 con 0x7f22f013f710 2024-03-25T18:39:17.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f00b2a30 con 0x7f22f01430c0 2024-03-25T18:39:17.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:17.512 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.510+0000 7f22f4923640 10 monclient(hunting): authenticate will time out at 7961.767773s 2024-03-25T18:39:17.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.519+0000 7f22ee59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f22f00a98f0 0x7f22f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54120/0 (socket says 172.21.15.55:54120) 2024-03-25T18:39:17.519 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.519+0000 7f22ee59c640 1 -- 172.21.15.55:0/1119843133 learned_addr learned my addr 172.21.15.55:0/1119843133 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:17.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.533+0000 7f22ed59a640 1 -- 172.21.15.55:0/1119843133 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354009532 0 0) 0x7f22f0142520 con 0x7f22f00a98f0 2024-03-25T18:39:17.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.533+0000 7f22ed59a640 10 monclient(hunting): _negotiate 2024-03-25T18:39:17.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.533+0000 7f22ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:17.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.533+0000 7f22ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:17.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.533+0000 7f22ed59a640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:39:17.534 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.533+0000 7f22ed59a640 1 -- 172.21.15.55:0/1119843133 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d8003520 con 0x7f22f00a98f0 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 1 -- 172.21.15.55:0/1119843133 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2360129056 0 0) 0x7f22d8003520 con 0x7f22f00a98f0 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 1 -- 172.21.15.55:0/1119843133 >> v1:172.21.15.55:6790/0 conn(0x7f22f01430c0 legacy=0x7f22f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 1 -- 172.21.15.55:0/1119843133 >> v1:172.21.15.133:6789/0 conn(0x7f22f013f710 legacy=0x7f22f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 10 monclient: _finish_hunting 0 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 1 monclient: found mon.noname-a 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 1 -- 172.21.15.55:0/1119843133 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f00b44c0 con 0x7f22f00a98f0 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 10 monclient: _finish_auth 0 2024-03-25T18:39:17.547 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 10 monclient: _check_auth_tickets 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 1 -- 172.21.15.55:0/1119843133 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f22e8001a10 con 0x7f22f00a98f0 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22f4923640 5 monclient: authenticate success, global_id 4737 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22f4923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22ed59a640 10 monclient: dump: 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:17.548 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:17.549 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:17.549 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:17.549 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.546+0000 7f22f4923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.565+0000 7f22ed59a640 1 -- 172.21.15.55:0/1119843133 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e8001b70 con 0x7f22f00a98f0 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.565+0000 7f22ed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.565+0000 7f22ed59a640 1 -- 172.21.15.55:0/1119843133 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f22e8001fc0 con 0x7f22f00a98f0 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.565+0000 7f22f4923640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.565+0000 7f22f4923640 4 set_mon_vals no callback set 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.565+0000 7f22ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.565+0000 7f22ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.565+0000 7f22ed59a640 10 monclient: dump: 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:17.566 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:17.567 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:17.567 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:17.567 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:17.567 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:17.567 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:17.567 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:17.567 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:17.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.566+0000 7f22f4923640 10 monclient: shutdown 2024-03-25T18:39:17.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.566+0000 7f22f4923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:17.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.566+0000 7f22f4923640 1 -- 172.21.15.55:0/1119843133 >> v1:172.21.15.55:6789/0 conn(0x7f22f00a98f0 legacy=0x7f22f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:17.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.566+0000 7f22f4923640 1 -- 172.21.15.55:0/1119843133 shutdown_connections 2024-03-25T18:39:17.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.566+0000 7f22f4923640 1 -- 172.21.15.55:0/1119843133 >> 172.21.15.55:0/1119843133 conn(0x7f22f00b7ea0 msgr2=0x7f22f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:17.569 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.566+0000 7f22f4923640 1 -- 172.21.15.55:0/1119843133 shutdown_connections 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.566+0000 7f22f4923640 1 -- 172.21.15.55:0/1119843133 wait complete. 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.567+0000 7f22f4923640 10 monclient: build_initial_monmap 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.567+0000 7f22f4923640 10 monclient: monmap: 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.567+0000 7f22f4923640 1 librados: starting msgr at 2024-03-25T18:39:17.570 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.567+0000 7f22f4923640 1 librados: starting objecter 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.567+0000 7f22f4923640 1 Processor -- start 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.567+0000 7f22f4923640 1 -- start start 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.567+0000 7f22f4923640 1 librados: setting wanted keys 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.567+0000 7f22f4923640 1 librados: calling monclient init 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.567+0000 7f22f4923640 10 monclient: init 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 10 monclient(hunting): picked mon.noname-a con 0x7f22f00a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 10 monclient(hunting): picked mon.noname-c con 0x7f22f013f710 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 10 monclient(hunting): picked mon.noname-b con 0x7f22f01430c0 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f00abd30 con 0x7f22f00a98f0 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f01d77a0 con 0x7f22f01430c0 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f01d51f0 con 0x7f22f013f710 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:17.571 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.568+0000 7f22f4923640 10 monclient(hunting): authenticate will time out at 7961.825626s 2024-03-25T18:39:17.576 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.575+0000 7f22ee59c640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f22f00a98f0 0x7f22f01d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:54188/0 (socket says 172.21.15.55:54188) 2024-03-25T18:39:17.576 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.575+0000 7f22ee59c640 1 -- 172.21.15.55:0/1685191659 learned_addr learned my addr 172.21.15.55:0/1685191659 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:17.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.576+0000 7f22deffd640 10 client.?.objecter ms_handle_connect 0x7f22f00a98f0 2024-03-25T18:39:17.577 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.576+0000 7f22deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:17.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.582+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407378452 0 0) 0x7f22f00abd30 con 0x7f22f00a98f0 2024-03-25T18:39:17.583 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.583+0000 7f22deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:17.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.583+0000 7f22deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:17.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.583+0000 7f22deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:17.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.583+0000 7f22deffd640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:39:17.584 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.583+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d0004eb0 con 0x7f22f00a98f0 2024-03-25T18:39:17.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.586+0000 7f22deffd640 10 client.?.objecter ms_handle_connect 0x7f22f01430c0 2024-03-25T18:39:17.587 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.586+0000 7f22deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:17.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.588+0000 7f22deffd640 10 client.?.objecter ms_handle_connect 0x7f22f013f710 2024-03-25T18:39:17.588 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.588+0000 7f22deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.592+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 863890284 0 0) 0x7f22d0004eb0 con 0x7f22f00a98f0 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.592+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f00abd30 con 0x7f22f00a98f0 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.592+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f22e8001f40 con 0x7f22f00a98f0 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.592+0000 7f22deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.592+0000 7f22deffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.592+0000 7f22deffd640 10 monclient(hunting): dump: 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:17.593 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:17.594 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:17.594 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:17.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.592+0000 7f22deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:17.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.594+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525045952 0 0) 0x7f22f01d77a0 con 0x7f22f01430c0 2024-03-25T18:39:17.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.594+0000 7f22deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:17.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.594+0000 7f22deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:17.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.594+0000 7f22deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:17.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.594+0000 7f22deffd640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:39:17.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.594+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d0004eb0 con 0x7f22f01430c0 2024-03-25T18:39:17.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.595+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115337593 0 0) 0x7f22f01d51f0 con 0x7f22f013f710 2024-03-25T18:39:17.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.595+0000 7f22deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:17.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.595+0000 7f22deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:17.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.595+0000 7f22deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:17.596 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.595+0000 7f22deffd640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:39:17.597 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.595+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f01d77a0 con 0x7f22f013f710 2024-03-25T18:39:17.599 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3621594142 0 0) 0x7f22f00abd30 con 0x7f22f00a98f0 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 >> v1:172.21.15.55:6790/0 conn(0x7f22f013f710 legacy=0x7f22f0138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 >> v1:172.21.15.133:6789/0 conn(0x7f22f01430c0 legacy=0x7f22f01e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 1 monclient: found mon.a 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f01d83d0 con 0x7f22f00a98f0 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 10 monclient: _finish_auth 0 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22f4923640 5 monclient: authenticate success, global_id 4740 2024-03-25T18:39:17.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22f4923640 10 monclient: _renew_subs 2024-03-25T18:39:17.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22f4923640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:17.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f0138a50 con 0x7f22f00a98f0 2024-03-25T18:39:17.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22f4923640 10 client.4740.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:17.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22f4923640 10 monclient: _renew_subs 2024-03-25T18:39:17.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22f4923640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:17.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f01e35c0 con 0x7f22f00a98f0 2024-03-25T18:39:17.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.598+0000 7f22f4923640 1 librados: init done 2024-03-25T18:39:17.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.599+0000 7f22f4923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:17.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.599+0000 7f22f4923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:17.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.599+0000 7f22f4923640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:17.602 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.599+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f00627b0 con 0x7f22f00a98f0 2024-03-25T18:39:17.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.602+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e8004f70 con 0x7f22f00a98f0 2024-03-25T18:39:17.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.602+0000 7f22deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:17.603 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.602+0000 7f22dffff640 4 set_mon_vals no callback set 2024-03-25T18:39:17.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.912+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f22e8005280 con 0x7f22f00a98f0 2024-03-25T18:39:17.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.912+0000 7f22deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:17.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.912+0000 7f22deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:17.913 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.912+0000 7f22deffd640 10 monclient: dump: 2024-03-25T18:39:17.913 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:17.913 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:17.913 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:17.914 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:17.914 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:17.914 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:17.914 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:17.914 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:17.914 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:17.914 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:17.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.913+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f22e801b250 con 0x7f22f00a98f0 2024-03-25T18:39:17.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.914+0000 7f22edd9b640 10 monclient: get_auth_request con 0x7f22d006de00 auth_method 0 2024-03-25T18:39:17.915 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.914+0000 7f22deffd640 10 client.4740.objecter ms_handle_connect 0x7f22d006de00 2024-03-25T18:39:17.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.919+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7f22e8054e70 con 0x7f22f00a98f0 2024-03-25T18:39:17.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.919+0000 7f22deffd640 10 client.4740.objecter ms_dispatch 0x7f22f0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:39:17.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.919+0000 7f22deffd640 3 client.4740.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:39:17.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.919+0000 7f22deffd640 3 client.4740.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:39:17.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.919+0000 7f22deffd640 20 client.4740.objecter dump_active .. 0 homeless 2024-03-25T18:39:17.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.919+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22e80507b0 con 0x7f22f00a98f0 2024-03-25T18:39:17.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.919+0000 7f22deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:17.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:17.919+0000 7f22deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:18.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.175+0000 7f22f4923640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:18.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.175+0000 7f22f4923640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:18.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.175+0000 7f22f4923640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:18.176 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.175+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22f01d8560 con 0x7f22f00a98f0 2024-03-25T18:39:18.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.197+0000 7f22deffd640 1 -- 172.21.15.55:0/1685191659 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7f22e80528b0 con 0x7f22f00a98f0 2024-03-25T18:39:18.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.197+0000 7f22deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:18.198 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.197+0000 7f22deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:18.198 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:39:18.198 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:18.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 10 librados: watch_flush enter 2024-03-25T18:39:18.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 10 librados: watch_flush exit 2024-03-25T18:39:18.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 20 client.4740.objecter shutdown clearing up homeless session... 2024-03-25T18:39:18.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 10 client.4740.objecter successfully canceled tick 2024-03-25T18:39:18.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 >> v1:172.21.15.55:6800/286536064 conn(0x7f22d006de00 legacy=0x7f22d00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:18.201 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 10 monclient: shutdown 2024-03-25T18:39:18.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:18.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 >> v1:172.21.15.55:6789/0 conn(0x7f22f00a98f0 legacy=0x7f22f01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:18.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 shutdown_connections 2024-03-25T18:39:18.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 >> 172.21.15.55:0/1685191659 conn(0x7f22f00b7ea0 msgr2=0x7f22f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:18.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 shutdown_connections 2024-03-25T18:39:18.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 1 -- 172.21.15.55:0/1685191659 wait complete. 2024-03-25T18:39:18.202 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:18.200+0000 7f22f4923640 1 librados: shutdown 2024-03-25T18:39:18.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:19.634 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:29.804 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:19.639 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:29.809 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:19.662 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:29.800 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:19.663 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:29.833 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:19.667 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:29.805 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:19.679 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:29.849 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:19.682 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:29.852 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:19.683 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:29.821 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:19.699 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:29.869 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:25.021 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:25.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.121+0000 7f770b50d640 10 monclient: get_monmap_and_config 2024-03-25T18:39:25.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.121+0000 7f770b50d640 10 monclient: build_initial_monmap 2024-03-25T18:39:25.122 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.121+0000 7f770b50d640 10 monclient: monmap: 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:39:25.122673+0000 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:39:25.122673+0000 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:39:25.123 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:25.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.122+0000 7f770b50d640 1 Processor -- start 2024-03-25T18:39:25.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.122+0000 7f770b50d640 1 -- start start 2024-03-25T18:39:25.124 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.122+0000 7f770b50d640 10 monclient: init 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 10 monclient(hunting): picked mon.noname-c con 0x7f77040b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 10 monclient(hunting): picked mon.noname-b con 0x7f77040b1730 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 10 monclient(hunting): picked mon.noname-a con 0x7f770414be90 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77040af940 con 0x7f770414be90 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77040b0a70 con 0x7f77040b4d60 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77040b0d50 con 0x7f77040b1730 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:25.125 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.123+0000 7f770b50d640 10 monclient(hunting): authenticate will time out at 7969.380663s 2024-03-25T18:39:25.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.137+0000 7f7709aaa640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7f770414be90 0x7f770414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:49560/0 (socket says 172.21.15.55:49560) 2024-03-25T18:39:25.137 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.137+0000 7f7709aaa640 1 -- 172.21.15.55:0/2295897865 learned_addr learned my addr 172.21.15.55:0/2295897865 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:25.164 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.163+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225540869 0 0) 0x7f77040af940 con 0x7f770414be90 2024-03-25T18:39:25.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.163+0000 7f76fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:25.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.163+0000 7f76fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:25.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.163+0000 7f76fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:25.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.163+0000 7f76fbfff640 10 monclient(hunting): my global_id is 4743 2024-03-25T18:39:25.165 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.163+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f0003520 con 0x7f770414be90 2024-03-25T18:39:25.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.179+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391194687 0 0) 0x7f77040b0a70 con 0x7f77040b4d60 2024-03-25T18:39:25.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.179+0000 7f76fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:25.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.179+0000 7f76fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:25.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.179+0000 7f76fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:25.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.179+0000 7f76fbfff640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:39:25.180 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.179+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77040af940 con 0x7f77040b4d60 2024-03-25T18:39:25.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 213730043 0 0) 0x7f76f0003520 con 0x7f770414be90 2024-03-25T18:39:25.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 >> v1:172.21.15.55:6790/0 conn(0x7f77040b1730 legacy=0x7f77040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:39:25.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 >> v1:172.21.15.133:6789/0 conn(0x7f77040b4d60 legacy=0x7f770413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:25.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:39:25.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 1 monclient: found mon.noname-a 2024-03-25T18:39:25.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:25.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.55:6789/0 2024-03-25T18:39:25.185 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77040b1030 con 0x7f770414be90 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 10 monclient: _finish_auth 0 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f76fc001a10 con 0x7f770414be90 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f770b50d640 5 monclient: authenticate success, global_id 4743 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f770b50d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f76fbfff640 10 monclient: dump: 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.186 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:25.187 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:25.187 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:25.187 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:25.187 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:25.187 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:25.187 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.184+0000 7f770b50d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:25.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.218+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 <== mon.0 v1:172.21.15.55:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76fc001b70 con 0x7f770414be90 2024-03-25T18:39:25.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.218+0000 7f76fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:25.219 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.218+0000 7f76fbfff640 1 -- 172.21.15.55:0/2295897865 <== mon.0 v1:172.21.15.55:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f76fc001f80 con 0x7f770414be90 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.218+0000 7f76fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.218+0000 7f76fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.218+0000 7f76fbfff640 10 monclient: dump: 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.218+0000 7f770b50d640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:25.220 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.218+0000 7f770b50d640 4 set_mon_vals no callback set 2024-03-25T18:39:25.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.219+0000 7f770b50d640 10 monclient: shutdown 2024-03-25T18:39:25.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.219+0000 7f770b50d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:25.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.219+0000 7f770b50d640 1 -- 172.21.15.55:0/2295897865 >> v1:172.21.15.55:6789/0 conn(0x7f770414be90 legacy=0x7f770414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:25.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.219+0000 7f770b50d640 1 -- 172.21.15.55:0/2295897865 shutdown_connections 2024-03-25T18:39:25.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.219+0000 7f770b50d640 1 -- 172.21.15.55:0/2295897865 >> 172.21.15.55:0/2295897865 conn(0x7f77040617a0 msgr2=0x7f7704061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:25.222 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.219+0000 7f770b50d640 1 -- 172.21.15.55:0/2295897865 shutdown_connections 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.219+0000 7f770b50d640 1 -- 172.21.15.55:0/2295897865 wait complete. 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.220+0000 7f770b50d640 10 monclient: build_initial_monmap 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.220+0000 7f770b50d640 10 monclient: monmap: 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.220+0000 7f770b50d640 1 librados: starting msgr at 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.220+0000 7f770b50d640 1 librados: starting objecter 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.220+0000 7f770b50d640 1 Processor -- start 2024-03-25T18:39:25.223 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 1 -- start start 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 1 librados: setting wanted keys 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 1 librados: calling monclient init 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 10 monclient: init 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 10 monclient(hunting): picked mon.noname-b con 0x7f77040b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 10 monclient(hunting): picked mon.noname-a con 0x7f77040b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 10 monclient(hunting): picked mon.noname-c con 0x7f770414be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7704141b80 con 0x7f77040b4d60 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7704138930 con 0x7f77040b1730 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7704142280 con 0x7f770414be90 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:25.224 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.221+0000 7f770b50d640 10 monclient(hunting): authenticate will time out at 7969.479010s 2024-03-25T18:39:25.231 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.231+0000 7f77092a9640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f77040b1730 0x7f7704138220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:52280/0 (socket says 172.21.15.55:52280) 2024-03-25T18:39:25.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.231+0000 7f77092a9640 1 -- 172.21.15.55:0/1596434919 learned_addr learned my addr 172.21.15.55:0/1596434919 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:25.232 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.232+0000 7f76f9ffb640 10 client.?.objecter ms_handle_connect 0x7f77040b1730 2024-03-25T18:39:25.233 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.232+0000 7f76f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:25.238 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.238+0000 7f76f9ffb640 10 client.?.objecter ms_handle_connect 0x7f770414be90 2024-03-25T18:39:25.239 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.238+0000 7f76f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:25.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.240+0000 7f76f9ffb640 10 client.?.objecter ms_handle_connect 0x7f77040b4d60 2024-03-25T18:39:25.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.240+0000 7f76f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:25.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.242+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040313437 0 0) 0x7f7704138930 con 0x7f77040b1730 2024-03-25T18:39:25.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.242+0000 7f76f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:25.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.242+0000 7f76f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:25.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.242+0000 7f76f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:25.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.242+0000 7f76f9ffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:39:25.243 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.242+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e8004eb0 con 0x7f77040b1730 2024-03-25T18:39:25.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.247+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137554778 0 0) 0x7f7704142280 con 0x7f770414be90 2024-03-25T18:39:25.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.247+0000 7f76f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:25.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.247+0000 7f76f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:25.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.247+0000 7f76f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:25.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.247+0000 7f76f9ffb640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:39:25.248 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.247+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7704138930 con 0x7f770414be90 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.254+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2954233287 0 0) 0x7f7704138930 con 0x7f770414be90 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.254+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7704142280 con 0x7f770414be90 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.254+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f76fc001e80 con 0x7f770414be90 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.254+0000 7f76f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.254+0000 7f76f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.254+0000 7f76f9ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:25.255 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:25.256 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:25.256 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:25.256 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:25.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.254+0000 7f76f9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:25.256 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297172617 0 0) 0x7f7704141b80 con 0x7f77040b4d60 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 10 monclient(hunting): my global_id is 4746 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7704138930 con 0x7f77040b4d60 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 340823466 0 0) 0x7f76e8004eb0 con 0x7f77040b1730 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.133:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7704141b80 con 0x7f77040b1730 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f7700001a90 con 0x7f77040b1730 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.255+0000 7f76f9ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:25.257 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:25.258 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.258 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.258 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:25.258 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:25.258 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:25.258 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:25.258 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:25.258 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:25.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 521531642 0 0) 0x7f7704142280 con 0x7f770414be90 2024-03-25T18:39:25.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 >> v1:172.21.15.133:6789/0 conn(0x7f77040b1730 legacy=0x7f7704138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:25.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 >> v1:172.21.15.55:6789/0 conn(0x7f77040b4d60 legacy=0x7f7704134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:25.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:39:25.264 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 1 monclient: found mon.c 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7704142410 con 0x7f770414be90 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f76f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f770b50d640 5 monclient: authenticate success, global_id 4727 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f770b50d640 10 monclient: _renew_subs 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f770b50d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7704140960 con 0x7f770414be90 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f770b50d640 10 client.4727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f770b50d640 10 monclient: _renew_subs 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f770b50d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77041356a0 con 0x7f770414be90 2024-03-25T18:39:25.265 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.263+0000 7f770b50d640 1 librados: init done 2024-03-25T18:39:25.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.264+0000 7f770b50d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:25.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.264+0000 7f770b50d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:25.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.264+0000 7f770b50d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:25.266 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.264+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7704140cb0 con 0x7f770414be90 2024-03-25T18:39:25.593 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.592+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76fc0015c0 con 0x7f770414be90 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.592+0000 7f76f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.592+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f76fc004e10 con 0x7f770414be90 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.592+0000 7f76f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.593+0000 7f76f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.593+0000 7f76f9ffb640 10 monclient: dump: 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:25.594 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:25.595 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:25.595 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:25.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.592+0000 7f76fb7fe640 4 set_mon_vals no callback set 2024-03-25T18:39:25.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.593+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f76fc01ade0 con 0x7f770414be90 2024-03-25T18:39:25.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.594+0000 7f77092a9640 10 monclient: get_auth_request con 0x7f76e806ded0 auth_method 0 2024-03-25T18:39:25.595 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.594+0000 7f76f9ffb640 10 client.4727.objecter ms_handle_connect 0x7f76e806ded0 2024-03-25T18:39:25.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.599+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7f76fc054a00 con 0x7f770414be90 2024-03-25T18:39:25.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.599+0000 7f76f9ffb640 10 client.4727.objecter ms_dispatch 0x7f7704000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:39:25.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.599+0000 7f76f9ffb640 3 client.4727.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:39:25.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.599+0000 7f76f9ffb640 3 client.4727.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:39:25.600 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.599+0000 7f76f9ffb640 20 client.4727.objecter dump_active .. 0 homeless 2024-03-25T18:39:25.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.599+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76fc050340 con 0x7f770414be90 2024-03-25T18:39:25.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.599+0000 7f76f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:25.601 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.599+0000 7f76f9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:25.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.863+0000 7f770b50d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:25.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.863+0000 7f770b50d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:25.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.863+0000 7f770b50d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:25.864 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.863+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77040627b0 con 0x7f770414be90 2024-03-25T18:39:25.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.892+0000 7f76f9ffb640 1 -- 172.21.15.55:0/1596434919 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7f76fc052440 con 0x7f770414be90 2024-03-25T18:39:25.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.892+0000 7f76f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:25.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.892+0000 7f76f9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:25.893 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:39:25.894 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:25.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 10 librados: watch_flush enter 2024-03-25T18:39:25.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 10 librados: watch_flush exit 2024-03-25T18:39:25.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 20 client.4727.objecter shutdown clearing up homeless session... 2024-03-25T18:39:25.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 10 client.4727.objecter successfully canceled tick 2024-03-25T18:39:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 >> v1:172.21.15.55:6800/286536064 conn(0x7f76e806ded0 legacy=0x7f76e8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 10 monclient: shutdown 2024-03-25T18:39:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 >> v1:172.21.15.55:6790/0 conn(0x7f770414be90 legacy=0x7f77041d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 shutdown_connections 2024-03-25T18:39:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 >> 172.21.15.55:0/1596434919 conn(0x7f77040617a0 msgr2=0x7f7704061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.895+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 shutdown_connections 2024-03-25T18:39:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.896+0000 7f770b50d640 1 -- 172.21.15.55:0/1596434919 wait complete. 2024-03-25T18:39:25.897 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:25.896+0000 7f770b50d640 1 librados: shutdown 2024-03-25T18:39:25.909 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:32.715 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:32.832 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.832+0000 7f658cf5d640 10 monclient: get_monmap_and_config 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.832+0000 7f658cf5d640 10 monclient: build_initial_monmap 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.832+0000 7f658cf5d640 10 monclient: monmap: 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:39:32.833007+0000 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:39:32.833007+0000 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:39:32.833 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.832+0000 7f658cf5d640 1 Processor -- start 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.832+0000 7f658cf5d640 1 -- start start 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.832+0000 7f658cf5d640 10 monclient: init 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.833+0000 7f658cf5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.833+0000 7f658cf5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.833+0000 7f658cf5d640 10 monclient(hunting): picked mon.noname-a con 0x7f65880b4d60 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.833+0000 7f658cf5d640 10 monclient(hunting): picked mon.noname-c con 0x7f65880b1730 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.833+0000 7f658cf5d640 10 monclient(hunting): picked mon.noname-b con 0x7f658814be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.833+0000 7f658cf5d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65880af940 con 0x7f65880b4d60 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.834+0000 7f658cf5d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65880b0a70 con 0x7f65880b1730 2024-03-25T18:39:32.835 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.834+0000 7f658cf5d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65880b0d50 con 0x7f658814be90 2024-03-25T18:39:32.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.834+0000 7f658cf5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:32.836 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.834+0000 7f658cf5d640 10 monclient(hunting): authenticate will time out at 7977.091379s 2024-03-25T18:39:32.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.843+0000 7f657dd9b640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7f65880b1730 0x7f65880b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:49300/0 (socket says 172.21.15.55:49300) 2024-03-25T18:39:32.843 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.843+0000 7f657dd9b640 1 -- 172.21.15.55:0/3847922954 learned_addr learned my addr 172.21.15.55:0/3847922954 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:32.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.855+0000 7f6585d9b640 1 -- 172.21.15.55:0/3847922954 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025948464 0 0) 0x7f65880b0a70 con 0x7f65880b1730 2024-03-25T18:39:32.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.856+0000 7f6585d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:39:32.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.856+0000 7f6585d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:32.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.856+0000 7f6585d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:32.856 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.856+0000 7f6585d9b640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:39:32.857 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.856+0000 7f6585d9b640 1 -- 172.21.15.55:0/3847922954 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6574003520 con 0x7f65880b1730 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 1 -- 172.21.15.55:0/3847922954 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179972700 0 0) 0x7f6574003520 con 0x7f65880b1730 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 1 -- 172.21.15.55:0/3847922954 >> v1:172.21.15.55:6790/0 conn(0x7f658814be90 legacy=0x7f658814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 1 -- 172.21.15.55:0/3847922954 >> v1:172.21.15.55:6789/0 conn(0x7f65880b4d60 legacy=0x7f658813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 1 monclient: found mon.noname-c 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 1 -- 172.21.15.55:0/3847922954 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65880b1030 con 0x7f65880b1730 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 10 monclient: _finish_auth 0 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 1 -- 172.21.15.55:0/3847922954 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6570001690 con 0x7f65880b1730 2024-03-25T18:39:32.867 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f6585d9b640 10 monclient: dump: 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f658cf5d640 5 monclient: authenticate success, global_id 4720 2024-03-25T18:39:32.868 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.865+0000 7f658cf5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:32.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f6585d9b640 1 -- 172.21.15.55:0/3847922954 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65700017f0 con 0x7f65880b1730 2024-03-25T18:39:32.885 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f6585d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f6585d9b640 1 -- 172.21.15.55:0/3847922954 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6570001c40 con 0x7f65880b1730 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f6585d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f6585d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f6585d9b640 10 monclient: dump: 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:32.886 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:32.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f658cf5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:32.887 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f658cf5d640 4 set_mon_vals no callback set 2024-03-25T18:39:32.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f658cf5d640 10 monclient: shutdown 2024-03-25T18:39:32.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f658cf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:32.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.885+0000 7f658cf5d640 1 -- 172.21.15.55:0/3847922954 >> v1:172.21.15.133:6789/0 conn(0x7f65880b1730 legacy=0x7f65880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.888 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.886+0000 7f658cf5d640 1 -- 172.21.15.55:0/3847922954 shutdown_connections 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.886+0000 7f658cf5d640 1 -- 172.21.15.55:0/3847922954 >> 172.21.15.55:0/3847922954 conn(0x7f65880617a0 msgr2=0x7f6588061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.886+0000 7f658cf5d640 1 -- 172.21.15.55:0/3847922954 shutdown_connections 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.886+0000 7f658cf5d640 1 -- 172.21.15.55:0/3847922954 wait complete. 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.886+0000 7f658cf5d640 10 monclient: build_initial_monmap 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.886+0000 7f658cf5d640 10 monclient: monmap: 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:32.889 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.887+0000 7f658cf5d640 1 librados: starting msgr at 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.887+0000 7f658cf5d640 1 librados: starting objecter 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.887+0000 7f658cf5d640 1 Processor -- start 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.887+0000 7f658cf5d640 1 -- start start 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.887+0000 7f658cf5d640 1 librados: setting wanted keys 2024-03-25T18:39:32.890 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.887+0000 7f658cf5d640 1 librados: calling monclient init 2024-03-25T18:39:32.891 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.887+0000 7f658cf5d640 10 monclient: init 2024-03-25T18:39:32.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:32.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:32.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 10 monclient(hunting): picked mon.noname-a con 0x7f65880b1730 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:32.892 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 10 monclient(hunting): picked mon.noname-b con 0x7f65880b4d60 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:32.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 10 monclient(hunting): picked mon.noname-c con 0x7f658814be90 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:32.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65881380a0 con 0x7f65880b1730 2024-03-25T18:39:32.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658812f7b0 con 0x7f65880b4d60 2024-03-25T18:39:32.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658813c730 con 0x7f658814be90 2024-03-25T18:39:32.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:32.893 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.888+0000 7f658cf5d640 10 monclient(hunting): authenticate will time out at 7977.145791s 2024-03-25T18:39:32.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.893+0000 7f6586d9d640 1 --1- >> v1:172.21.15.55:6790/0 conn(0x7f658814be90 0x7f658812f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6790/0 says I am v1:172.21.15.55:46274/0 (socket says 172.21.15.55:46274) 2024-03-25T18:39:32.894 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.893+0000 7f6586d9d640 1 -- 172.21.15.55:0/3824346436 learned_addr learned my addr 172.21.15.55:0/3824346436 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:32.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.895+0000 7f657f7fe640 10 client.?.objecter ms_handle_connect 0x7f658814be90 2024-03-25T18:39:32.896 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.895+0000 7f657f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.899+0000 7f657f7fe640 10 client.?.objecter ms_handle_connect 0x7f65880b4d60 2024-03-25T18:39:32.899 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.899+0000 7f657f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:32.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.902+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139655814 0 0) 0x7f658813c730 con 0x7f658814be90 2024-03-25T18:39:32.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.903+0000 7f657f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:32.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.903+0000 7f657f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:32.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.903+0000 7f657f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:32.903 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.903+0000 7f657f7fe640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:39:32.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.903+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6568004eb0 con 0x7f658814be90 2024-03-25T18:39:32.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.903+0000 7f657f7fe640 10 client.?.objecter ms_handle_connect 0x7f65880b1730 2024-03-25T18:39:32.904 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.903+0000 7f657f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.916+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279245993 0 0) 0x7f65881380a0 con 0x7f65880b1730 2024-03-25T18:39:32.916 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.916+0000 7f657f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:32.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.916+0000 7f657f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:32.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.916+0000 7f657f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:32.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.916+0000 7f657f7fe640 10 monclient(hunting): my global_id is 4752 2024-03-25T18:39:32.917 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.916+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658813c730 con 0x7f65880b1730 2024-03-25T18:39:32.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.918+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 394515966 0 0) 0x7f6568004eb0 con 0x7f658814be90 2024-03-25T18:39:32.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.918+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 --> v1:172.21.15.55:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65881380a0 con 0x7f658814be90 2024-03-25T18:39:32.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.918+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6580001c30 con 0x7f658814be90 2024-03-25T18:39:32.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.918+0000 7f657f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:32.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.918+0000 7f657f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:32.919 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.918+0000 7f657f7fe640 10 monclient(hunting): dump: 2024-03-25T18:39:32.919 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:32.919 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:32.920 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:32.920 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:32.920 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:32.920 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:32.920 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:32.920 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:32.920 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:32.920 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:32.920 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.918+0000 7f657f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:32.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.919+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288363132 0 0) 0x7f658812f7b0 con 0x7f65880b4d60 2024-03-25T18:39:32.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.919+0000 7f657f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:32.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.919+0000 7f657f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:32.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.919+0000 7f657f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:32.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.919+0000 7f657f7fe640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:39:32.921 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.919+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6568004eb0 con 0x7f65880b4d60 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1471416933 0 0) 0x7f65881380a0 con 0x7f658814be90 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 >> v1:172.21.15.133:6789/0 conn(0x7f65880b4d60 legacy=0x7f658812e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 >> v1:172.21.15.55:6789/0 conn(0x7f65880b1730 legacy=0x7f6588135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 1 monclient: found mon.c 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 --> v1:172.21.15.55:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6588139120 con 0x7f658814be90 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f657f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f658cf5d640 5 monclient: authenticate success, global_id 4733 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f658cf5d640 10 monclient: _renew_subs 2024-03-25T18:39:32.927 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f658cf5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 --> v1:172.21.15.55:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65881457a0 con 0x7f658814be90 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f658cf5d640 10 client.4733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f658cf5d640 10 monclient: _renew_subs 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f658cf5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 --> v1:172.21.15.55:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6588136620 con 0x7f658814be90 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.925+0000 7f658cf5d640 1 librados: init done 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.926+0000 7f658cf5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.926+0000 7f658cf5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.926+0000 7f658cf5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:32.928 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:32.926+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65881392b0 con 0x7f658814be90 2024-03-25T18:39:33.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.239+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6580003990 con 0x7f658814be90 2024-03-25T18:39:33.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.239+0000 7f657f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:33.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.239+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7f6580003c00 con 0x7f658814be90 2024-03-25T18:39:33.240 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.239+0000 7f657f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.239+0000 7f657f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.239+0000 7f657f7fe640 10 monclient: dump: 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:33.241 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.239+0000 7f658559a640 4 set_mon_vals no callback set 2024-03-25T18:39:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.240+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7f6580003e20 con 0x7f658814be90 2024-03-25T18:39:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.241+0000 7f658659c640 10 monclient: get_auth_request con 0x7f656806de00 auth_method 0 2024-03-25T18:39:33.242 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.241+0000 7f657f7fe640 10 client.4733.objecter ms_handle_connect 0x7f656806de00 2024-03-25T18:39:33.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.245+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7f65800547f0 con 0x7f658814be90 2024-03-25T18:39:33.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.245+0000 7f657f7fe640 10 client.4733.objecter ms_dispatch 0x7f6588000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:39:33.246 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.245+0000 7f657f7fe640 3 client.4733.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:39:33.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.245+0000 7f657f7fe640 3 client.4733.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:39:33.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.246+0000 7f657f7fe640 20 client.4733.objecter dump_active .. 0 homeless 2024-03-25T18:39:33.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.246+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6580050300 con 0x7f658814be90 2024-03-25T18:39:33.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.246+0000 7f657f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:33.247 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.246+0000 7f657f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:33.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.504+0000 7f658cf5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:33.506 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.504+0000 7f658cf5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:33.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.504+0000 7f658cf5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.55:6790/0 2024-03-25T18:39:33.507 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.504+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 --> v1:172.21.15.55:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65880afbd0 con 0x7f658814be90 2024-03-25T18:39:33.688 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.686+0000 7f657f7fe640 1 -- 172.21.15.55:0/3824346436 <== mon.2 v1:172.21.15.55:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7f6580052230 con 0x7f658814be90 2024-03-25T18:39:33.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.686+0000 7f657f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:33.689 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.686+0000 7f657f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:33.689 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:39:33.689 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:33.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 10 librados: watch_flush enter 2024-03-25T18:39:33.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 10 librados: watch_flush exit 2024-03-25T18:39:33.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 20 client.4733.objecter shutdown clearing up homeless session... 2024-03-25T18:39:33.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 10 client.4733.objecter successfully canceled tick 2024-03-25T18:39:33.692 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 >> v1:172.21.15.55:6800/286536064 conn(0x7f656806de00 legacy=0x7f65680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:33.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 10 monclient: shutdown 2024-03-25T18:39:33.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:33.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 >> v1:172.21.15.55:6790/0 conn(0x7f658814be90 legacy=0x7f658812f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:33.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 shutdown_connections 2024-03-25T18:39:33.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 >> 172.21.15.55:0/3824346436 conn(0x7f65880617a0 msgr2=0x7f65880b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:33.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 shutdown_connections 2024-03-25T18:39:33.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 1 -- 172.21.15.55:0/3824346436 wait complete. 2024-03-25T18:39:33.693 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:33.689+0000 7f658cf5d640 1 librados: shutdown 2024-03-25T18:39:33.703 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:36.920 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:47.090 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:36.972 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:47.142 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:37.013 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:11:47.183 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:37.038 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:47.208 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:37.041 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:47.211 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:37.064 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:11:47.234 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:37.068 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:47.206 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:37.075 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:47.213 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:37.092 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:11:47.230 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:40.510 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:40.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.627+0000 7faf5ed67640 10 monclient: get_monmap_and_config 2024-03-25T18:39:40.629 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.627+0000 7faf5ed67640 10 monclient: build_initial_monmap 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.627+0000 7faf5ed67640 10 monclient: monmap: 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:39:40.629789+0000 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:39:40.629789+0000 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-c 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-b 2024-03-25T18:39:40.630 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:40.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.628+0000 7faf5ed67640 1 Processor -- start 2024-03-25T18:39:40.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.628+0000 7faf5ed67640 1 -- start start 2024-03-25T18:39:40.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.628+0000 7faf5ed67640 10 monclient: init 2024-03-25T18:39:40.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:40.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:40.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 10 monclient(hunting): picked mon.noname-a con 0x7faf580a98f0 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:40.634 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 10 monclient(hunting): picked mon.noname-c con 0x7faf5813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:40.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 10 monclient(hunting): picked mon.noname-b con 0x7faf581430c0 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:40.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf58142520 con 0x7faf580a98f0 2024-03-25T18:39:40.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf580b28a0 con 0x7faf5813f710 2024-03-25T18:39:40.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf580b2a30 con 0x7faf581430c0 2024-03-25T18:39:40.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:40.635 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.629+0000 7faf5ed67640 10 monclient(hunting): authenticate will time out at 7984.887997s 2024-03-25T18:39:40.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.648+0000 7faf4ffff640 1 --1- >> v1:172.21.15.133:6789/0 conn(0x7faf5813f710 0x7faf581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.133:6789/0 says I am v1:172.21.15.55:41354/0 (socket says 172.21.15.55:41354) 2024-03-25T18:39:40.649 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.648+0000 7faf4ffff640 1 -- 172.21.15.55:0/3523354881 learned_addr learned my addr 172.21.15.55:0/3523354881 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:40.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.660+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097205231 0 0) 0x7faf580b28a0 con 0x7faf5813f710 2024-03-25T18:39:40.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.660+0000 7faf4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:40.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.660+0000 7faf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:40.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.660+0000 7faf4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:40.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.660+0000 7faf4f7fe640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:39:40.662 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.660+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf40003520 con 0x7faf5813f710 2024-03-25T18:39:40.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.676+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771205223 0 0) 0x7faf58142520 con 0x7faf580a98f0 2024-03-25T18:39:40.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.676+0000 7faf4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:40.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.676+0000 7faf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:40.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.676+0000 7faf4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:40.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.676+0000 7faf4f7fe640 10 monclient(hunting): my global_id is 4755 2024-03-25T18:39:40.678 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.676+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf580b28a0 con 0x7faf580a98f0 2024-03-25T18:39:40.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.678+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 <== mon.2 v1:172.21.15.55:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556940781 0 0) 0x7faf580b2a30 con 0x7faf581430c0 2024-03-25T18:39:40.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.678+0000 7faf4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:40.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.678+0000 7faf4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:40.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.678+0000 7faf4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:40.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.678+0000 7faf4f7fe640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:39:40.680 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.678+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 --> v1:172.21.15.55:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf58142520 con 0x7faf581430c0 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 <== mon.1 v1:172.21.15.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816721028 0 0) 0x7faf40003520 con 0x7faf5813f710 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 >> v1:172.21.15.55:6790/0 conn(0x7faf581430c0 legacy=0x7faf5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 >> v1:172.21.15.55:6789/0 conn(0x7faf580a98f0 legacy=0x7faf580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.133:6789/0 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 --> v1:172.21.15.133:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf580b44c0 con 0x7faf5813f710 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 <== mon.1 v1:172.21.15.133:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7faf48001690 con 0x7faf5813f710 2024-03-25T18:39:40.699 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf5ed67640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf5ed67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf4f7fe640 10 monclient: dump: 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:40.700 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.697+0000 7faf5ed67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:40.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 <== mon.1 v1:172.21.15.133:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf480017f0 con 0x7faf5813f710 2024-03-25T18:39:40.716 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf4f7fe640 1 -- 172.21.15.55:0/3523354881 <== mon.1 v1:172.21.15.133:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7faf48001c00 con 0x7faf5813f710 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf4f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf4f7fe640 10 monclient: dump: 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf5ed67640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:40.717 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf5ed67640 4 set_mon_vals no callback set 2024-03-25T18:39:40.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf5ed67640 10 monclient: shutdown 2024-03-25T18:39:40.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf5ed67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:40.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.715+0000 7faf5ed67640 1 -- 172.21.15.55:0/3523354881 >> v1:172.21.15.133:6789/0 conn(0x7faf5813f710 legacy=0x7faf581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:40.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.716+0000 7faf5ed67640 1 -- 172.21.15.55:0/3523354881 shutdown_connections 2024-03-25T18:39:40.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.716+0000 7faf5ed67640 1 -- 172.21.15.55:0/3523354881 >> 172.21.15.55:0/3523354881 conn(0x7faf580b7ea0 msgr2=0x7faf580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:40.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.716+0000 7faf5ed67640 1 -- 172.21.15.55:0/3523354881 shutdown_connections 2024-03-25T18:39:40.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.716+0000 7faf5ed67640 1 -- 172.21.15.55:0/3523354881 wait complete. 2024-03-25T18:39:40.719 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.717+0000 7faf5ed67640 10 monclient: build_initial_monmap 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.717+0000 7faf5ed67640 10 monclient: monmap: 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:epoch 0 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:last_changed 0.000000 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:created 0.000000 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 1 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.noname-a 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.noname-b 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.noname-c 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.717+0000 7faf5ed67640 1 librados: starting msgr at 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.717+0000 7faf5ed67640 1 librados: starting objecter 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.717+0000 7faf5ed67640 1 Processor -- start 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.717+0000 7faf5ed67640 1 -- start start 2024-03-25T18:39:40.720 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.717+0000 7faf5ed67640 1 librados: setting wanted keys 2024-03-25T18:39:40.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.717+0000 7faf5ed67640 1 librados: calling monclient init 2024-03-25T18:39:40.721 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.717+0000 7faf5ed67640 10 monclient: init 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 10 monclient(hunting): picked mon.noname-c con 0x7faf580a98f0 addr v1:172.21.15.55:6790/0 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 10 monclient(hunting): picked mon.noname-b con 0x7faf5813f710 addr v1:172.21.15.133:6789/0 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 10 monclient(hunting): picked mon.noname-a con 0x7faf581430c0 addr v1:172.21.15.55:6789/0 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 1 -- --> v1:172.21.15.55:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf58138c10 con 0x7faf581430c0 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 1 -- --> v1:172.21.15.133:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf58138e60 con 0x7faf5813f710 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 1 -- --> v1:172.21.15.55:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf58139140 con 0x7faf580a98f0 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:40.722 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.718+0000 7faf5ed67640 10 monclient(hunting): authenticate will time out at 7984.977028s 2024-03-25T18:39:40.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.730+0000 7faf5d304640 1 --1- >> v1:172.21.15.55:6789/0 conn(0x7faf581430c0 0x7faf581e34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.55:6789/0 says I am v1:172.21.15.55:48968/0 (socket says 172.21.15.55:48968) 2024-03-25T18:39:40.732 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.730+0000 7faf5d304640 1 -- 172.21.15.55:0/3862532798 learned_addr learned my addr 172.21.15.55:0/3862532798 (peer_addr_for_me v1:172.21.15.55:0/0) 2024-03-25T18:39:40.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.736+0000 7faf4d7fa640 10 client.?.objecter ms_handle_connect 0x7faf581430c0 2024-03-25T18:39:40.738 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.736+0000 7faf4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:40.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.741+0000 7faf4d7fa640 10 client.?.objecter ms_handle_connect 0x7faf5813f710 2024-03-25T18:39:40.743 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.741+0000 7faf4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:40.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.745+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667817073 0 0) 0x7faf58138c10 con 0x7faf581430c0 2024-03-25T18:39:40.746 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.745+0000 7faf4d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:39:40.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.745+0000 7faf4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:40.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.745+0000 7faf4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:40.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.745+0000 7faf4d7fa640 10 monclient(hunting): my global_id is 4758 2024-03-25T18:39:40.747 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.745+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 --> v1:172.21.15.55:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf44004eb0 con 0x7faf581430c0 2024-03-25T18:39:40.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.746+0000 7faf4d7fa640 10 client.?.objecter ms_handle_connect 0x7faf580a98f0 2024-03-25T18:39:40.748 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.746+0000 7faf4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:40.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.750+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.1 v1:172.21.15.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031141894 0 0) 0x7faf58138e60 con 0x7faf5813f710 2024-03-25T18:39:40.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.750+0000 7faf4d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:39:40.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.750+0000 7faf4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:40.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.750+0000 7faf4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:40.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.750+0000 7faf4d7fa640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:39:40.752 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.750+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 --> v1:172.21.15.133:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf58138c10 con 0x7faf5813f710 2024-03-25T18:39:40.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.751+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2906203504 0 0) 0x7faf44004eb0 con 0x7faf581430c0 2024-03-25T18:39:40.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.752+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 --> v1:172.21.15.55:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf58138e60 con 0x7faf581430c0 2024-03-25T18:39:40.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.752+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7faf50001c70 con 0x7faf581430c0 2024-03-25T18:39:40.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.752+0000 7faf4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:40.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.752+0000 7faf4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:40.753 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.752+0000 7faf4d7fa640 10 monclient(hunting): dump: 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:40.754 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.752+0000 7faf4d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:40.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4043816721 0 0) 0x7faf58138e60 con 0x7faf581430c0 2024-03-25T18:39:40.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 >> v1:172.21.15.55:6790/0 conn(0x7faf580a98f0 legacy=0x7faf58137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:40.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 >> v1:172.21.15.133:6789/0 conn(0x7faf5813f710 legacy=0x7faf58132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:40.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:39:40.762 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 1 monclient: found mon.a 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 --> v1:172.21.15.55:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf58139920 con 0x7faf581430c0 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf5ed67640 5 monclient: authenticate success, global_id 4758 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf5ed67640 10 monclient: _renew_subs 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf5ed67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 --> v1:172.21.15.55:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf581382e0 con 0x7faf581430c0 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf5ed67640 10 client.4758.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf5ed67640 10 monclient: _renew_subs 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf5ed67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 --> v1:172.21.15.55:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf580ab8b0 con 0x7faf581430c0 2024-03-25T18:39:40.763 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.760+0000 7faf5ed67640 1 librados: init done 2024-03-25T18:39:40.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.761+0000 7faf5ed67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:40.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.761+0000 7faf5ed67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:40.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.761+0000 7faf5ed67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:40.764 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:40.761+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf58137a20 con 0x7faf581430c0 2024-03-25T18:39:41.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.070+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf50003970 con 0x7faf581430c0 2024-03-25T18:39:41.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.070+0000 7faf4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:41.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.070+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4206223956 0 0) 0x7faf50003d80 con 0x7faf581430c0 2024-03-25T18:39:41.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.070+0000 7faf4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:41.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.070+0000 7faf4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:41.072 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.070+0000 7faf4d7fa640 10 monclient: dump: 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:epoch 1 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:fsid a9b133ab-c902-44c5-bfee-b6d6f33f1d56 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:last_changed 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:created 2024-03-25T18:27:13.657292+0000 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:election_strategy: 3 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:0: v1:172.21.15.55:6789/0 mon.a 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:1: v1:172.21.15.133:6789/0 mon.b 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:2: v1:172.21.15.55:6790/0 mon.c 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr: 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.070+0000 7faf4effd640 4 set_mon_vals no callback set 2024-03-25T18:39:41.073 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.070+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 479873192 0 0) 0x7faf50003fa0 con 0x7faf581430c0 2024-03-25T18:39:41.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.071+0000 7faf5cb03640 10 monclient: get_auth_request con 0x7faf4406dd10 auth_method 0 2024-03-25T18:39:41.074 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.071+0000 7faf4d7fa640 10 client.4758.objecter ms_handle_connect 0x7faf4406dd10 2024-03-25T18:39:41.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.076+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3519750761 0 0) 0x7faf50054ac0 con 0x7faf581430c0 2024-03-25T18:39:41.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.076+0000 7faf4d7fa640 10 client.4758.objecter ms_dispatch 0x7faf58000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:39:41.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.076+0000 7faf4d7fa640 3 client.4758.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:39:41.078 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.076+0000 7faf4d7fa640 3 client.4758.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:39:41.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.076+0000 7faf4d7fa640 20 client.4758.objecter dump_active .. 0 homeless 2024-03-25T18:39:41.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.076+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf500505d0 con 0x7faf581430c0 2024-03-25T18:39:41.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.076+0000 7faf4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:41.079 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.076+0000 7faf4d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:41.332 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.330+0000 7faf5ed67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:41.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.330+0000 7faf5ed67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:41.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.330+0000 7faf5ed67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.55:6789/0 2024-03-25T18:39:41.333 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.330+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 --> v1:172.21.15.55:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf5813d010 con 0x7faf581430c0 2024-03-25T18:39:41.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.419+0000 7faf4d7fa640 1 -- 172.21.15.55:0/3862532798 <== mon.0 v1:172.21.15.55:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3356546074) 0x7faf50052500 con 0x7faf581430c0 2024-03-25T18:39:41.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.419+0000 7faf4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:41.421 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.419+0000 7faf4d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:41.421 INFO:teuthology.misc.health.smithi055.stdout: 2024-03-25T18:39:41.421 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"a9b133ab-c902-44c5-bfee-b6d6f33f1d56","created":"2024-03-25T18:28:12.401177+0000","modified":"2024-03-25T18:38:16.599845+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:38:16.599845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c847a581-38f4-4880-a9a7-ec6492573f17","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b0db4fe3-7bc7-4836-8bf4-c9744981c68c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"2effb575-2dc6-44ea-bc99-c85b6c87643c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"6d01598c-3cbe-4167-a7e5-606491d1eac8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fb85e8a1-9dab-45ae-8e58-20ea74da1dac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f181ab39-94f0-43d3-a755-cc9640c4f2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb2ff4a2-2840-4539-b1d5-76c0d5c7ddac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"11e302d1-2e5b-46e5-9ef8-09c7b65c028a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:41.423 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 10 librados: watch_flush enter 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 10 librados: watch_flush exit 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 20 client.4758.objecter shutdown clearing up homeless session... 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 10 client.4758.objecter successfully canceled tick 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 >> v1:172.21.15.55:6800/286536064 conn(0x7faf4406dd10 legacy=0x7faf440701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 10 monclient: shutdown 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 >> v1:172.21.15.55:6789/0 conn(0x7faf581430c0 legacy=0x7faf581e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 shutdown_connections 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 >> 172.21.15.55:0/3862532798 conn(0x7faf580b7ea0 msgr2=0x7faf58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 shutdown_connections 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 1 -- 172.21.15.55:0/3862532798 wait complete. 2024-03-25T18:39:41.424 INFO:teuthology.misc.health.smithi055.stderr:2024-03-25T18:39:41.421+0000 7faf5ed67640 1 librados: shutdown 2024-03-25T18:39:41.437 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:41.437 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:39:41.439 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:39:41.439 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:39:41.439 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:39:43.322 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:11:25.542 35191== 2024-03-25T18:39:43.322 INFO:tasks.ceph.osd.0.smithi055.stderr:==00:00:11:25.542 35191== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:39:43.362 INFO:tasks.ceph.osd.0.smithi055.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:39:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:39:47.541 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:39:47.542 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:39:47.542 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:39:47.542 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:39:49.405 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:11:31.600 35193== 2024-03-25T18:39:49.405 INFO:tasks.ceph.osd.1.smithi055.stderr:==00:00:11:31.600 35193== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:39:49.445 INFO:tasks.ceph.osd.1.smithi055.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:39:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:39:53.644 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:39:53.644 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:39:53.645 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:39:53.645 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:39:55.628 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:11:37.828 35192== 2024-03-25T18:39:55.628 INFO:tasks.ceph.osd.2.smithi055.stderr:==00:00:11:37.828 35192== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:39:55.676 INFO:tasks.ceph.osd.2.smithi055.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:39:57.006 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:12:07.176 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:57.013 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:12:07.183 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:57.040 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:12:07.210 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:57.061 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:07.198 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:57.065 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:07.203 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:57.087 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:12:07.257 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:57.091 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:12:07.261 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:57.094 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:07.232 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:57.115 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:12:07.285 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:39:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:39:59.746 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:39:59.747 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:39:59.747 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:39:59.747 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:01.697 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:11:43.892 35194== 2024-03-25T18:40:01.697 INFO:tasks.ceph.osd.3.smithi055.stderr:==00:00:11:43.892 35194== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:40:01.739 INFO:tasks.ceph.osd.3.smithi055.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:40:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:40:05.849 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:40:05.849 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:40:05.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:40:05.850 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:07.779 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:11:49.991 35430== 2024-03-25T18:40:07.779 INFO:tasks.ceph.osd.4.smithi133.stderr:==00:00:11:49.991 35430== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:40:07.815 INFO:tasks.ceph.osd.4.smithi133.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:40:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:40:11.952 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:40:11.952 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:40:11.953 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:40:11.953 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:13.958 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:11:56.167 35431== 2024-03-25T18:40:13.959 INFO:tasks.ceph.osd.5.smithi133.stderr:==00:00:11:56.167 35431== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:40:13.997 INFO:tasks.ceph.osd.5.smithi133.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:40:17.190 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:12:27.360 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:17.192 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:12:27.362 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:17.212 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:12:27.382 34876== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:17.238 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:27.376 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:17.239 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:12:27.409 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:17.241 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:27.379 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:17.244 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:12:27.415 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:17.258 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:27.395 34885== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:17.261 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:12:27.431 34881== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:40:18.054 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:40:18.055 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:40:18.055 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:40:18.055 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:19.747 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:12:01.951 35432== 2024-03-25T18:40:19.748 INFO:tasks.ceph.osd.6.smithi133.stderr:==00:00:12:01.951 35432== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:40:19.784 INFO:tasks.ceph.osd.6.smithi133.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:40:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:40:24.157 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:40:24.158 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:40:24.158 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:40:24.158 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:25.714 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:12:07.897 35433== 2024-03-25T18:40:25.824 INFO:tasks.ceph.osd.7.smithi133.stderr:==00:00:12:07.897 35433== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:40:25.825 INFO:tasks.ceph.osd.7.smithi133.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:40:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:40:30.260 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:40:30.261 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:40:30.261 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:40:30.261 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:40:30.261 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:30.262 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:40:30.261+0000 7fa1dee0e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 34886) UID: 0 2024-03-25T18:40:30.263 INFO:tasks.ceph.mgr.y.smithi055.stderr:2024-03-25T18:40:30.261+0000 7fa1dee0e640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:40:30.290 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:40:30.290 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:40:30.290 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:30.317 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:40:30.317 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:40:30.318 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:40:30.318 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:30.346 INFO:tasks.ceph.mon.a.smithi055.stderr:2024-03-25T18:40:30.333+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34846) UID: 0 2024-03-25T18:40:30.347 INFO:tasks.ceph.mon.a.smithi055.stderr:2024-03-25T18:40:30.335+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:40:30.845 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:12:41.015 34876== 2024-03-25T18:40:30.846 INFO:tasks.ceph.mon.a.smithi055.stderr:==00:00:12:41.015 34876== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:40:30.855 INFO:tasks.ceph.mon.a.smithi055.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:40:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:40:36.420 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:40:36.420 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:40:36.420 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:40:36.420 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:36.437 INFO:tasks.ceph.mon.c.smithi055.stderr:2024-03-25T18:40:36.423+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34848) UID: 0 2024-03-25T18:40:36.437 INFO:tasks.ceph.mon.c.smithi055.stderr:2024-03-25T18:40:36.425+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:40:42.808 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:12:52.978 34881== 2024-03-25T18:40:42.808 INFO:tasks.ceph.mon.c.smithi055.stderr:==00:00:12:52.978 34881== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:40:42.818 INFO:tasks.ceph.mon.c.smithi055.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:40:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:40:48.623 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:40:48.624 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:40:48.624 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:40:48.624 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:40:48.640 INFO:tasks.ceph.mon.b.smithi133.stderr:2024-03-25T18:40:48.628+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34881) UID: 0 2024-03-25T18:40:48.641 INFO:tasks.ceph.mon.b.smithi133.stderr:2024-03-25T18:40:48.629+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:40:49.062 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:59.199 34885== 2024-03-25T18:40:49.062 INFO:tasks.ceph.mon.b.smithi133.stderr:==00:00:12:59.199 34885== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:40:49.070 INFO:tasks.ceph.mon.b.smithi133.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:40:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:40:54.727 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:40:54.727 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:40:54.728 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:40:54.728 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:40:54.777 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:40:54.777 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:40:54.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:40:54.904 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:40:54.979 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:40:54.979 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:40:55.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:40:55.055 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:40:55.137 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:40:55.137 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:40:55.297 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:40:55.297 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:40:55.397 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:40:55.397 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:40:55.461 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:40:55.461 DEBUG:teuthology.orchestra.run.smithi133:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:40:55.511 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:40:55.513 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/data/mon.a.tgz 2024-03-25T18:40:55.513 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-25T18:40:55.530 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.HE5DcLA5o8 2024-03-25T18:40:55.530 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.HE5DcLA5o8 2024-03-25T18:40:55.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.HE5DcLA5o8 2024-03-25T18:40:55.713 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.HE5DcLA5o8 is 133KB 2024-03-25T18:40:55.767 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.HE5DcLA5o8 2024-03-25T18:40:55.782 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/data/mon.c.tgz 2024-03-25T18:40:55.782 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2024-03-25T18:40:55.836 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.2TfK6fYJar 2024-03-25T18:40:55.836 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.2TfK6fYJar 2024-03-25T18:40:55.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.2TfK6fYJar 2024-03-25T18:40:56.012 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.2TfK6fYJar is 133KB 2024-03-25T18:40:56.066 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.2TfK6fYJar 2024-03-25T18:40:56.080 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/data/mon.b.tgz 2024-03-25T18:40:56.080 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2024-03-25T18:40:56.098 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.UrgEVjoGeS 2024-03-25T18:40:56.098 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.UrgEVjoGeS 2024-03-25T18:40:56.199 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.UrgEVjoGeS 2024-03-25T18:40:56.286 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.UrgEVjoGeS is 133KB 2024-03-25T18:40:56.338 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.UrgEVjoGeS 2024-03-25T18:40:56.353 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:40:56.377 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:40:56.380 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:40:56.715 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:40:56.983 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:40:56.985 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/remote/smithi055/crash 2024-03-25T18:40:56.986 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:40:57.014 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/remote/smithi133/crash 2024-03-25T18:40:57.015 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:40:57.041 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:40:57.042 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:40:57.058 DEBUG:teuthology.orchestra.run.smithi133:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:40:57.126 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:40:57.126 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35280.log 2024-03-25T18:40:57.126 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:40:57.126 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.35280.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr: 72.1% 87.6% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz -- replaced with /var/log/ceph/ceph-client.admin.35280.log.gz 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/valgrind/osd.4.log: 69.5% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35080.log 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.35200.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.35080.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-03-25T18:40:57.127 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35080.log.gz 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35040.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.35040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35322.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35040.log.gz 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35240.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.35322.log: /var/log/ceph/ceph-client.admin.35240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35120.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35322.log.gz/var/log/ceph/ceph-mgr.x.log: 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35240.log.gz 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-client.admin.35120.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35160.log 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35120.log.gz/var/log/ceph/ceph.log: 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.35160.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:40:57.128 INFO:teuthology.orchestra.run.smithi133.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35160.log.gz 2024-03-25T18:40:57.129 INFO:teuthology.orchestra.run.smithi133.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41462.log 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41813.log 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41462.log.gz 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40135.log 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39471.log 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41032.log 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41813.log.gz 2024-03-25T18:40:57.136 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40135.log.gz 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39040.log 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39471.log.gz 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr: 86.4%/var/log/ceph/ceph-client.admin.41110.log: -- replaced with /var/log/ceph/ceph-client.admin.41032.log.gz 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41071.log 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41891.log 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41071.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39040.log.gz 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41423.log 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41071.log.gz 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41891.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41423.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41891.log.gz 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41423.log.gz 2024-03-25T18:40:57.137 INFO:teuthology.orchestra.run.smithi055.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29037.log 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41189.log 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph.tmp-client.admin.29037.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29037.log.gz 2024-03-25T18:40:57.138 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40369.log 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39276.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41189.log.gz 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40876.log 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40369.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39628.log 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40369.log.gz 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39784.log 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40174.log 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40876.log.gz 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39667.log 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39628.log.gz 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40018.log 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39784.log.gz 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2024-03-25T18:40:57.139 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40174.log.gz 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39667.log.gz 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34888.log 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41774.log 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40018.log.gz 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.34888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41618.log 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39354.log 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41774.log.gz 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39549.log 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41618.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.34888.log.gz 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38804.log 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41618.log.gz 2024-03-25T18:40:57.140 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41696.log 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38804.log: gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39549.log.gz -5 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37124.log 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41696.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38804.log.gz 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.37124.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41696.log.gz 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41384.log 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37124.log.gz 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose/var/log/ceph/ceph.audit.log: -- /var/log/ceph/ceph-client.admin.40954.log 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39706.log 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40954.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41384.log.gz 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39745.log 2024-03-25T18:40:57.141 INFO:teuthology.orchestra.run.smithi055.stderr: 86.9%/var/log/ceph/ceph-client.admin.39706.log: -- replaced with /var/log/ceph/ceph-client.admin.40954.log.gz 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41501.log 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40447.log 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39706.log.gz 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41735.log 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39745.log.gz 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40447.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41501.log.gz 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39118.log 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42008.log 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40447.log.gz 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41306.log 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41735.log.gz 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39118.log.gz 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40603.log 2024-03-25T18:40:57.142 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40057.log 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42008.log.gz 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41306.log.gz 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5%/var/log/ceph/ceph-client.admin.40057.log: -- replaced with /var/log/ceph/ceph-client.admin.40603.log.gz 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39589.log 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41267.log 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40057.log.gz 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41267.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39589.log.gz 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35225.log 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41267.log.gz 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35225.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-03-25T18:40:57.143 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40837.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.35225.log.gz 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39510.log 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36197.log 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39510.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39901.log 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39079.log 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.36197.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39510.log.gz 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39237.log 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36197.log.gz 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39079.log: gzip -5 --verbose 85.5% -- /var/log/ceph/ceph-client.admin.41969.log -- replaced with /var/log/ceph/ceph-client.admin.39901.log.gz 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39237.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39079.log.gz 2024-03-25T18:40:57.144 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40096.log 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr: 86.1%/var/log/ceph/ceph-client.admin.41969.log: -- replaced with /var/log/ceph/ceph-client.admin.39237.log.gz 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40798.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39159.log 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40681.log 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41228.log 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40096.log.gz 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40759.log 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39159.log.gz 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41228.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40681.log.gz 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38962.log 2024-03-25T18:40:57.145 INFO:teuthology.orchestra.run.smithi055.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41228.log.gz 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.3.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40759.log.gz 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39001.log 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41345.log 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38962.log: /var/log/ceph/ceph-client.admin.39001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39315.log 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38962.log.gz 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41345.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39001.log.gz 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41345.log.gz 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40330.log 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40408.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39315.log.gz 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40993.log 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-03-25T18:40:57.146 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38843.log 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40330.log.gz 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40993.log: gzip -5 --verbose 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr: -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41579.log 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40993.log.gz 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38843.log.gz 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40213.log 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41852.log: gzip -5 --verbose 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41579.log.gz 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr: -- /var/log/ceph/ceph-client.admin.42047.log 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40213.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42047.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:40:57.147 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-osd.0.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42047.log.gz 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39979.log 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39979.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40915.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39979.log.gz 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35411.log 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-osd.2.log: 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40564.log 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.35411.log: /var/log/ceph/ceph-client.admin.40564.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35411.log.gz 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38884.log 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.40291.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40564.log.gz 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39823.log 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42086.log 2024-03-25T18:40:57.148 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39823.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38884.log.gz 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41930.log 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39823.log.gz 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.42086.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42086.log.gz 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39940.log 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39940.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41930.log.gz 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39862.log 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39432.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39940.log.gz 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38482.log 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38482.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39862.log.gz 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41657.log 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.39198.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38482.log.gz 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-03-25T18:40:57.149 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41540.log 2024-03-25T18:40:57.150 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38923.log 2024-03-25T18:40:57.150 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.41540.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41657.log.gz 2024-03-25T18:40:57.150 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:40:57.150 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-client.admin.38923.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41540.log.gz 2024-03-25T18:40:57.150 INFO:teuthology.orchestra.run.smithi055.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38923.log.gz 2024-03-25T18:40:57.191 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph-mon.c.log: 92.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:40:57.942 INFO:teuthology.orchestra.run.smithi055.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:40:58.006 INFO:teuthology.orchestra.run.smithi133.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:40:58.019 INFO:teuthology.orchestra.run.smithi133.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:40:58.246 INFO:teuthology.orchestra.run.smithi055.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:40:58.284 INFO:teuthology.orchestra.run.smithi133.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:40:58.294 INFO:teuthology.orchestra.run.smithi133.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:40:58.318 INFO:teuthology.orchestra.run.smithi055.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:40:58.331 INFO:teuthology.orchestra.run.smithi055.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:41:02.161 INFO:teuthology.orchestra.run.smithi055.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:41:02.175 INFO:teuthology.orchestra.run.smithi055.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:41:02.177 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-03-25T18:41:02.177 INFO:teuthology.orchestra.run.smithi055.stderr:real 0m5.106s 2024-03-25T18:41:02.177 INFO:teuthology.orchestra.run.smithi055.stderr:user 0m14.176s 2024-03-25T18:41:02.177 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m0.546s 2024-03-25T18:41:02.202 INFO:teuthology.orchestra.run.smithi133.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:41:02.204 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-25T18:41:02.204 INFO:teuthology.orchestra.run.smithi133.stderr:real 0m5.107s 2024-03-25T18:41:02.204 INFO:teuthology.orchestra.run.smithi133.stderr:user 0m8.941s 2024-03-25T18:41:02.204 INFO:teuthology.orchestra.run.smithi133.stderr:sys 0m0.343s 2024-03-25T18:41:02.204 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:41:02.204 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/remote/smithi055/log 2024-03-25T18:41:02.205 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:41:04.554 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/remote/smithi133/log 2024-03-25T18:41:04.555 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:41:06.111 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:41:06.324 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fc3d701c7f8e4a0b94af1485cd009e06 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:41:06.328 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:41:06.337 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:41:06.353 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:41:06.354 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:41:06.368 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:41:06.449 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:41:06.458 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:41:06.459 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:41:06.462 DEBUG:teuthology.orchestra.run.smithi133:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:41:06.475 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:41:06.479 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:41:06.481 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:41:06.481 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-03-25T18:41:06.481 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +480us[ +480us] +/- 78ms 2024-03-25T18:41:06.481 INFO:teuthology.orchestra.run.smithi055.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -1055us[-1043us] +/- 63ms 2024-03-25T18:41:06.481 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +778us[ +778us] +/- 68ms 2024-03-25T18:41:06.482 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:06.486 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:41:06.486 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2024-03-25T18:41:06.486 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +456us[ +456us] +/- 78ms 2024-03-25T18:41:06.486 INFO:teuthology.orchestra.run.smithi133.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -1082us[-1078us] +/- 63ms 2024-03-25T18:41:06.486 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +758us[ +763us] +/- 68ms 2024-03-25T18:41:06.487 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:41:06.487 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:41:06.499 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:41:06.500 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:41:06.508 DEBUG:teuthology.orchestra.run.smithi055:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:41:06.595 DEBUG:teuthology.orchestra.run.smithi133:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:41:06.666 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:41:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:41:06.698 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:41:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:41:06.729 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:41:06.740 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:41:06.748 INFO:teuthology.task.internal:Duration was 7752.008232 seconds 2024-03-25T18:41:06.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:41:06.757 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:41:06.757 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:41:06.760 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:41:06.802 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:41:06.813 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:41:07.130 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:41:07.131 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi055.front.sepia.ceph.com 2024-03-25T18:41:07.131 DEBUG:teuthology.orchestra.run.smithi055:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:41:07.152 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi133.front.sepia.ceph.com 2024-03-25T18:41:07.152 DEBUG:teuthology.orchestra.run.smithi133:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:41:07.181 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:41:07.181 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:41:07.195 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:41:07.251 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:41:07.251 DEBUG:teuthology.orchestra.run.smithi055:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:41:07.265 DEBUG:teuthology.orchestra.run.smithi133:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:41:07.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:41:07.406 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:41:07.406 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:41:07.433 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:41:07.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:41:07.471 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:41:07.476 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:41:07.498 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2024-03-25T18:41:07.525 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2024-03-25T18:41:07.546 DEBUG:teuthology.orchestra.run.smithi055:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:41:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:41:07.579 DEBUG:teuthology.orchestra.run.smithi133:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:41:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:41:07.600 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:41:07.610 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:41:07.611 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/remote/smithi055 2024-03-25T18:41:07.611 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:41:07.655 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283/remote/smithi133 2024-03-25T18:41:07.655 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:41:07.691 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:41:07.691 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:41:07.693 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:41:07.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:41:07.754 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:41:07.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:41:07.769 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:41:07.770 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:41:07.772 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:41:07.784 INFO:teuthology.orchestra.run.smithi055.stdout: 265309 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:41 /home/ubuntu/cephtest 2024-03-25T18:41:07.795 INFO:teuthology.orchestra.run.smithi133.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:41 /home/ubuntu/cephtest 2024-03-25T18:41:07.796 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:41:07.809 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:41:07.839 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:41:07.868 DEBUG:teuthology.nuke:shortname: smithi055 2024-03-25T18:41:07.868 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:41:07.891 DEBUG:teuthology.nuke:shortname: smithi133 2024-03-25T18:41:07.892 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:41:07.895 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:23:19.371011', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK07BMq4oLcI6wwv4DMUGtscYrh9LL59zIyHfosL9Nj8Ex27kI0DSaI4yISwPX7nrZdnKhC/CUVAG8gzxGw/l/A='} 2024-03-25T18:41:07.920 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621283', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:23:19.372598', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqMAS6+p/Y9IuVWSK3DNUKE1ozDOdJ65gM1d/iplZtXgOL9VzTNrOGN+qGVjwh4Pt4gCuRRJXRbtcLOhe0ylkA='} 2024-03-25T18:41:07.941 INFO:teuthology.orchestra.console.smithi055:Power off 2024-03-25T18:41:07.942 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:41:07.967 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-03-25T18:41:07.967 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:41:07.985 INFO:teuthology.orchestra.console.smithi133:Power off 2024-03-25T18:41:07.986 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:41:07.996 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-25T18:41:08.010 DEBUG:teuthology.orchestra.console.smithi133:power off output: Chassis Power Control: Down/Off 2024-03-25T18:41:08.010 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:41:08.033 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2024-03-25T18:41:11.997 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:41:12.034 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:41:12.109 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-25T18:41:12.147 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2024-03-25T18:41:16.114 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:41:16.148 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:41:16.224 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-25T18:41:16.259 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2024-03-25T18:41:20.226 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:41:20.261 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:41:20.337 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-25T18:41:20.371 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is off 2024-03-25T18:41:20.439 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-03-25T18:41:20.472 INFO:teuthology.orchestra.console.smithi133:Power off completed 2024-03-25T18:41:20.577 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7752.008232355118 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fc3d701c7f8e4a0b94af1485cd009e06 status: fail success: false 2024-03-25T18:41:20.578 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:41:20.655 INFO:teuthology.run:FAIL